With "on the contrary" you oppose what is said, you say the fact is the opposite of what is said. On the other hand, yet still, that said (that being said), however, nontheless and some others sound better to me.

(Added) In your example, it's hard to tell without more context. I think you should write "On the other hand, there may be some who would agree with you."

Suppose the first alternative is X. "On the contrary" refers to a choice which is the exact opposite of X. "On the other hand" refers a alternative choice which may not be the exact opposite of X. Don't use "on the contrary" unless you mean "the exact opposite." Here are some examples of use. 1) "On the one hand, we could take the George Washington Bridge. On the other hand, we could take the Tappan Zee Bridge." The two bridges are alternatives, not opposites. 2) "I need to turn right." "On the contrary, you need to turn left." The two choices are exact opposites.

This reminds me of Sherlock Holmes. He often replied to Dr Watson, when disagreeing with him, "On the contrary, my dear fellow." 'On the other hand' is, in my opinion, a more common way of suggesting another point of view. On the one hand, I think this. On the other, I think that.

Yes it is okay to use it