Despite working hard, he still failed the exam. It was too hard for him. Despite working very little, he still passed the exam. It was easy for him. Despite + noun phrase or gerund phrase which defines something which makes no difference to the situation in the main clause. He wasn't lazy. To / on the contrary, he was the hardest-working student in the class. (opposite) This is a complete phrase and is followed by a comma

You can't use these two expressions in the same sentence. You have to choose one or the other. "Despite" must be followed by a noun or a noun phrase. Despite Henry's satisfaction with the weather in England, he fondly recalls the dry climate in Russia. "To/on the contrary" introduces something that you wouldn't expect, given the previous sentence. Henry thought it would be very dry in England. On the contrary, it is very rainy there.

Despite Adams's self-effacing claims to the contrary, Henry is an effective, innovative teacher and editor. Despite Henry likes the weather in England to the contrary, he recalls the dry climite in Russia.

Give us a little more context. There are many ways this could go. Give an example or 2 so we know how you'd like to use it. . on the contrary?

It depends on the kind of sentence. The word despite can be a preposition and contrary can be an adjective or noun.