The grammar is: noun + to the contrary You just have to learn this expression as a whole. Perhaps a historian of the English language could tell you why "to" was chosen but it wouldn't help you to learn the expression. Trying to find consistent rules in expressions like this one is probably not going to help you. My guess is that the syntax comes from French (au contraire) or Latin (modern Italian - al contrario). When you want to refer to a whole idea (not just a noun), "On the contrary, ...." is more common.