In casual speech in Canada, people rarely make a distinction. I agree with John's explanation, "come with me" and "go with person X."
I can give examples:
#1 go
Me: I'm going to the mall.
My wife: Josh, you wanted to buy a shirt at the mall.
You should go with your father.
[= You should go away from here to the mall.]
#2 go [I don't use this version but some of my friends do.]
Me: I'm going to the mall. Josh, you wanted to buy a shirt at the mall.
You should go with me.
[= You should go away from here to the mall.]
#3 come [I use this version.]
Me: I'm going to the mall. Josh, you wanted to buy a shirt at the mall.
You should come with me.
[= You should come here (to me) and then we will go to the mall together.]