"I'm off too" implies you are going some where. "I'm going to" can mean you are going to do something or go somewhere. For example, if ever you are going to see a wizard you can say "I'm off to". We're off to see the wizard, The Wonderful Wizard of Oz We hear he is a whiz of a wiz, if ever a wiz there was.

I means "I'm leaving right now for a party" but in an informal way.

I think it means= I am leaving, I am going somewhere, as you said.