When you look forward to you look forward to something. The gerund is replacing a noun.

The infinitive includes the word "to" - to cook, to drive, to be. I want to cook dinner now. He knows how to drive. "to" is also used as a preposition. I go to the shopping mall. Your rule that "to" is followed by a infinitive is not a good rule. Where did you learn this rule?

"Looking forward to" is an idiomatic phrasal verb and "hearing from you" is the direct object of the phrasal verb. 'To' is a preposition particle and is not looking for an object of a preposition. It is part of the verb phrase 'looking forward to ' and not referring to an infinitive.