In casual speech in Canada, people rarely make a distinction. I agree with John's explanation, "come with me" and "go with person X." I can give examples: #1 go Me: I'm going to the mall. My wife: Josh, you wanted to buy a shirt at the mall. You should go with your father. [= You should go away from here to the mall.] #2 go [I don't use this version but some of my friends do.] Me: I'm going to the mall. Josh, you wanted to buy a shirt at the mall. You should go with me. [= You should go away from here to the mall.] #3 come [I use this version.] Me: I'm going to the mall. Josh, you wanted to buy a shirt at the mall. You should come with me. [= You should come here (to me) and then we will go to the mall together.]

Go = to go away from a location or person. Come = come to; or towards a location or person We would say go with them/him/her/that man/women/ person the teacher/your teacher/your aunt/my father etc any third party person. We would only general say come with me, to ask/request command someone to come with you, or to ask request command that they bring something with them, or come escorted by somebody else etc.. "come fully equipped" ""come with your elder brother it will be dark at that time of night"

Go with me makes no sense. Come with me makes perfect sense :).