People say both, although the correct form is (oddly enough) "he's a friend of my brother's." This is more obvious when we use a pronoun "he's a friend of mine." No native speaker would ever say "a friend of *me."

He's a friend of my brother. He's my brother's friend. He's a friend of my brother's (what?). --> this sounds like an unfinished sentence to me.