"Can I just ignore the "h" sounds in whatever the case is?" No. Absolutely no. This would mean that you're mispronouncing words.
As others have mentioned, certain words have a silent H and you just need to learn them. Of course, in contractions (have -> 've) then the H disappears. And yes, there are dialects which drop the H, but as a learner of English you need to focus on the English that everyone understands: standard English.
I think what you're asking is when you can connect a word starting with H to the preceding word. You might know that our plosive sounds (P T K) actually have a puff of air (H) following them, so a phrase such as "stop him" or "meet her" might sound like we're dropping the H, but we're still pronouncing it. Basically, "stop him" and "stop-im" do sound different.
There's no pronunciation trick here. Just learn to speak smoothly and naturally, and check with natve and advanced speakers to see if you're sounding right.