As a native speaker I was never confused about the ㅁ sound, but I have heard this same comment. So it seems to sound close to "b" sometimes. Given that "m" and "b" sounds are made with almost the same formation of the lips, I suppose it is understandable that they can sometimes sound similar. I doubt however that there is any special trick to help with the situation other than knowing a lot of words and phrases. Native speakers are not confused because they already know all the words and phrases so they can usually predict what might or might not be said in a given situation. Likewise, as you learn more words and phrases, I think this confusion will slowly go away as you get better at guessing what might be said.