The direction of 来 is towards the speaker.
The direction of 去 is against the speaker.
Jimmy got a phone call when we were playing football this morning. It was his girlfriend. 'Yes, honey?', Jimmy answered the phone. 'When will you 回来?', asked Abby; meanwhile, Jimmy smilingly gave the finger to my wry face and went, 'I will 回去 in one or two hours.'