Mind if I have a go? :)
Ok, to me the basic idea is that going=leaving, and coming=arriving. However, as fdmaxey points out, it depends on where the speaker is. Here's a couple of additional points:
When you say "I'm going (home, somewhere)", the implication is that you're leaving where you are, or leaving the company you are with, even if you don't actually state that.
When using "to come", it also implies that a meeting will take place, or you'll be together with people. Eg. when you say "are you coming to the beach?" it suggests that your friend will meet with you. When you say "are you going to the beach?", you have to add "...with us?" to include yourself/selves.
Good question! It's one of those tricky uses of English which is common but not easily explained.