It sounds unnatural. I left right away. I left immediately. I didn't hang around at the gym. I left right away.

DavidK has it right..."I didn't hang around" is probably the most natural way to express what you want to say, especially in North American English.

I mean grammatically that's not incorrect. But you didn't really answer his question. He asked "why were you late to get back home from the gym" not what you did after the gym. So, he expected you to be at home earlier. Usually I would say: 1) after I was done, I didn't wait a single moment to rush back home. 2) I didn't wait a single moment to get back home after I was done. "I didn't wait a single moment" + to do what + rest...