"Go back" is something you were to say if you were already somewhere else.
For example: "I need to go back home because I forgot my bag."
"Come back" would be used if you're talking to someone who is at a different place.
For example: "It is past your curfew, you need to come back home."
"Come" would refer to near the speaker, so you wouldn't say "I need to come back home." unless you left home and were talking to someone who is back home.
I'm kind of over-complicating it, but I think that's the general basis behind it.