It could be the opposite of "hold your breath" which figuratively means to wait or delay until something special happens: " expect the mail to be delivered soon, but I'm not holding my breath. It's often late." "Don't hold your breath, I gotta go out now." could be in a situation where the 'unknown' caller is telling you not to be surprised upon his leaving out from somewhere or something. it's like: 'RELAX, I'll be safe upon leaving / going out. I gotta move now." :)

It means not to expect an event to happen soon. "He said he was going to wash the car, but don't hold your breath." You may have to wait a long time for a clean car. The phrase is an exaggeration of the fact that you would not be able to hold your breath (stop breathing) while waiting for him to wash the car.

In that context, they're saying, "Don't wait for me, I'm going out now and I won't be back anytime soon."

Don't depend on it! Your enemy says invites you to the cinema and to meet her there at 10. You tell her not to hold her breath (you doubt that you will go and she shouldn't expect you there). Normally said under the breath - quietly - because the other person is not really supposed to hear you say it! It is sarcastic and a little rude.