To "look down on (someone)" is the correct term in this context. You should only use "look down at (someone)" if you mean that the person is literally looking downward, because they're standing in a higher position. To "look down on" someone is to have a low opinion of them. (It sounds like the book author made a mistake here. Or, if this is a line of dialogue in a story, perhaps the author intended for the character to use the phrase incorrectly.) (By the way, is there a Mandarin equivalent for this? In Cantonese it's 睇起, but it looks like that's not used in Mandarin.)

As well as "Look down on you", in practice you will also see "Look down at you" written and spoken (with the same meaning intended). The first is the 'technical' correct version, but I would not call the 2nd incorrect as you will commonly see it written and spoken by native speakers. For me, for what it's worth, the 2nd sounds fine. It's possible usage varies from country to country.

"Look down on you" doesn't sound right. I'm pretty sure that everyone would notice the difference and find it odd.