You are right about the 'French Connection'. Inconsistencies in pronunciation often come from the various points in history at which these words were absorbed into the English language. The same goes for words which end in 'age'. The French words which have been part of the English language for many centuries acquired an unstressed 'English' pronunciation (like 'village'), while later imports retained their French pronunciation (like 'sabotage') , with some words (like 'garage') falling somewhere in between, with a somewhat undecided pronunciation. As for the silent 'h', it's only 'heir', 'hour' and 'honour/honor' and its derivatives (honesty, honest, honorable etc) where the 'h' is never pronounced. In US English, 'herb' is sometimes pronounced 'erb', but not in GB English. There was a time when 'hotel' and 'historic' also had silent initial letters ('an hotel' and 'an historic' event' ), but this has almost entirely fallen out of use in modern-day English.

One thought: For some words you do pronounce the H when using the word alone, such as "hotel", but it becomes silent when used in a sentence, "an hotel". Perhaps people rarely used the individual form so the sentence form became the standard?

There's also heirloom, heir, heiress, haute couture, and haute cuisine

I can think of honest, hour and heir. Why that is, I don't know. English had quite a renaissance when many French words got imported, so that could easily be the reason.