By definitions standards, they are very similar.  That are almost the same words, even. 

Traitor is the most common and less formal.  You will use this almost always.

Betrayer is more poetic and much more formal; regal, almost.   

For example: 

John the traitor, is a description and that is all  

John the betrayer, is less of a description and more of a title and adds a poetic sound to it indicating something of grandeur.  Something that everyone would know the reason for.

Though it doesn’t sound right to directly call someone a betrayer, it isn’t incorrect.  In those cases you would usually use the verb to betray (I betrayed the king) or traitor.  

Today, it is usually only heard as “the betrayer”  in a title