This is a very casual phrase used in informal spoken language. To bum someone out means to disappoint someone, or to make someone unhappy, usually by an unexpected negative reaction, or by giving them unwelcome information. Example: A: Great news: I got the job at the zoo! B: You do know that the last person who took that job got eaten by a tiger, don't you? A. Hey, don't tell me that! You're bumming me out. Here are some other ways of saying the same thing: You're getting me down. You're disappointing me. You're making me feel depressed. You're making me feel bad. A 'bummer' is a disappointment, a piece of bad luck, or a bad experience. You may hear someone say, "Man, that's a real bummer!", or sometimes just the single word, "Bummer!", in response to hearing bad news from a friend. This is a very informal word used in casual spoken English. The same idea can be expressed in standard English as: That's too bad. That's a pity. That's a shame. That is disappointing. That is bad luck.

You could say "You annoying me"

You're driving me crazy; you're depressing me