Your sentence uses "you" three times but the middle "you" is a different person from the first and third "you". This is confusing, so let's change your sentence to fix this: (*) "To hear you speak, one would think you were one of the privileged few." You could also use "I" or "we" rather than "one". In (*), the phrase "to hear you speak" is an adjectival phrase that modifies "one". The meaning of (*) is the same as the meaning of "Hearing you speak, one would think you were one of the privileged few" and, indeed, this is a simpler way of saying the same thing. Infinitive phrases can be difficult to interpret. They can act as adjectives, nouns, or adverbs. With experience will you learn to recognize how they are acting. Here are some similar examples in which "to hear you speak" acts in various roles: NOUN: "To hear you speak, one would think to be a great pleasure." ADVERB: "To hear you speak, one would need a hearing aid." ADJECTIVE: "To hear you speak, one would think you were one of the privileged few."

To hear you speak - sounds more formal / educated. "when you are speaking" - less formal / everyday speech.

This is my view, in terms of British English. I use "to hear you speak" when I am commenting on the content of what the other person is saying: their opinions, etc. Using "when you are speaking" I am either talking about the person's way of speaking, their tone of voice, etc, or the effect they have "everybody pays attention", "nothing else matters to me", "the hairs on the back of my neck stand up", etc ... In your example "to hear you speak": the opinions, etc the person is giving suggest they are one of the privileged few; "when you are speaking" their accent, or the authority, confidence, arrogance, etc in their voice suggest they are one of the privileged few. HTH