I rarely use 'whom' for the sole reason that it is uncommon and creates a formal distance between speakers.
Unfortunately, many people do not understand the history of language and grammar. There are 2 types of grammars: prescriptive and descriptive. Prescriptive grammar defines how a language is "supposed" to be used. Descriptive grammar is how grammar is actually used. Let me give you a very, very brief history of the English language:
Prior to the year 1066 (the year of the Battle of Hastings), English had 4 grammatical cases, one of which was the accusative. (Whom is an 'accusative' case. Meaning it is the object of a verb or preposition.) But as English developed due to contact with French, and as is the trend for all Indo-European languages, English lost cases in favor of using prepositions. We still have vestiges of the accusative case (him, her, me), but even some these are falling out of favor in many dialects of English. There is not one English language. There are multiple Englishes. This is a basic fact of English Linguistics.
And there was no such thing as a standard English grammar before the 16th century. 16th century grammarians wanted to make English seem more regal, so they arbitrarily took rules from Latin grammar and tacked them on to English even though no native speakers actually talked like this.
It is an unfortunate fact that the "correct" (prescriptive) way to speak English always happens to be the way people with economic and political power speak it (*cough* white people *cough*). However, it can easily be proven with linguistic grammar (generative phrase-structure modeling) that many communities of English speakers do not use words like "whom," and yet somehow they all still completely understand each other. How can you tell an entire community of speakers that they are using their own language incorrectly?
People insist that those of us who routinely drop words like "whom" from our vocabularies are uneducated. This simply isn't true.