I think you can say both, I mean neither would be a "grammatical mistake" in the real meaning of the word. I think you should just "prefer" one infinite or the other in accordance with either of the following situations:
If you say: "SIN AVISARME", it means that you thought, or that "ustedes estaban de acuerdo" that they should inform you "AT THE SAME MOMENT THAT THEY WERE GOING AWAY". In this case, the 2 facts, "informing you" and "going away", would have been CONTEMPORARY, so it would be preferable to say "SIN AVISARME", because this infinitive is not "antecedent" to "se ayan ido". For instance, they could have phoned you saying: "We are going away!" at the very moment they were going out of the door. That could have been your previous "agreement". You could have told them: "Phone me to inform me at the moment you're going away".
On the other hand, if you thought, or if "ustedes estaban de acuerdo" that they should inform you "A CERTAIN TIME BEFORE THEY WENT AWAY" (maybe hours before, or even the day before), in this case it would be preferable to say "SIN HABERME AVISADO", since the past infinitive underlines the "antecedence" of "INFORMING YOU" compared to "GOING AWAY".