Tom: "Where are you going? The party has just started."
John: "I'm off [going] to work now. Today I work a late shift because someone called in sick."
The expression "off to work" reminds me of a well-known song from the classic children's movie "The Wizard of Oz": "We're off to see the wizard, the Wonderful Wizard of Oz. We hear he is a whiz of a wiz, if ever a wiz there was..."