I mean grammatically that's not incorrect.
But you didn't really answer his question.
He asked "why were you late to get back home from the gym" not what you did after the gym.
So, he expected you to be at home earlier.
Usually I would say:
1) after I was done, I didn't wait a single moment to rush back home.
2) I didn't wait a single moment to get back home after I was done.
"I didn't wait a single moment" + to do what + rest...