When you are using the term to mean someone said offensive words, you would use "at."
"He swore at me."
When you are using it to mean that someone made a promise, you use "to."
"He swore to me that he would never leave me."
Of course, as Randy showed, there are other ways to use "swear" without prepositions or as a phrasal verb.