Most of the speakers of US can absolutely tell the difference just by listening, because most US speakers say a hard "R" sound. In part of the US like Massachusetts, they may speak the two words more similarly.
I tell my students to get the hard "r" to put their teeth together, and say rrr, rrr, almost like an angry dog. This will help you to say the word farther differently than father. Good luck, or try to schedule with me and I will practice this with you.