tuhruk is right.
"~ wo yobu" always comes with a person or its equivalent, like police, teacher, whom you can call.
"~wo yomu" always comes with something you can read such as a book, newspaper, website, magazine, etc.
Besides that, when we actually say them in conversation, yonde for yobu and yonde for yomu are pronounced differently.
Oh, also, they are different in kanji. So if it's written, there is no way to mistake.
For pronunciation, it's difficult to show the exact pronunciation by text, but this is my best try:
H means High pitch, and L means Low pitch.
H L - H L L
Yobu - Yonde
L H - L H H