Basically, it's whether the word is followed by an action or an idea.
You convince someone to do something. That's an action.
You convince someone of something. That's an idea.
Check these examples:
I tried to convince her (to believe) that I could fly.
-You don't need to say "to believe" because it's understood, but the action is there.
I tried to convince her of my ability to fly.
-This also shows that it can be awkward to use one over the other sometimes. The first sentence sounds much more natural than the second.