It could be the opposite of "hold your breath" which figuratively means to wait or delay until something special happens:
" expect the mail to be delivered soon, but I'm not holding my breath. It's often late."
"Don't hold your breath, I gotta go out now." could be in a situation where the 'unknown' caller is telling you not to be surprised upon his leaving out from somewhere or something.
it's like: 'RELAX, I'll be safe upon leaving / going out. I gotta move now."