This is how I understand it:
1. In 99% of the cases, only "He is not a friend of MINE" would be correct.
2. But there are (rare) occasions when "He is not a friend of me" would be correct:
Let's say that you are the president of a club. Mr. X. does not like your leadership. Thus, you might tell a friend that "Mr. X is not a friend of ME." That is, he does not like your policies. It has nothing to do with personal friendship.