The meaning changes if you remove "me". It's still a legitimate sentence, but it's not the one you started with.
If you say, "He is not only annoying me but also my brother," then you're specifying that he is specifically an annoyance to you (POSSIBLY: read below for an explanation on why this may not be the case). 'Annoying', in that sentence, is a verb.
If you say, "He is not only annoying but also my brother," then you're now saying that being annoying is a general trait of his: that he is an annoyance to EVERYONE at all times instead of simply being annoying to you for a temporary period. 'Annoying', in that sentence, is an adjective.
Of course, in either case, I'd recommend making it a compound sentence. While what you wrote is acceptable, it's also weird-sounding. And it's very ambiguous if you include 'me', because you can also read it as "He is not only annoying me, but he's also annoying my brother" instead of "He's not only annoying me, but he is also my brother". Both of those interpretations are entirely legitimate and could only be deduced through context (something that a sentence like this likely would not have). By putting a comma before the 'but' and fully forming a second sentence, you can specify whether 'he' is the man annoying you and your brother OR whether 'he' IS your annoying brother.