I would use it is really a hassle instead of saying its bothering me. You can say its bothering, irritating or annoying to express your feelings. You can say it is annoying when someone asks you to do something you don't want to do. But I would tell the person in a nicer manner than saying any of these. I would say Im a bit busy and cant do whatever you are being asked to do. But If you were talking about the matter to a close friend or family you can be more upfront and say that what you are asking me to do is bothering me because I dont want to do it. Basically they are interchangeable when expressing emotions, Someone can bother, annoy, irritate you all at the same time for the same reason. But irritate can also mean something that is causing discomfort, physically and emotionally. Annoying would be used to express how you feel about a certain situation. If someone keeps prank calling you it can be annoying that someone is bothering you. and bother can be used to say Sorry To bother you, or you can tell someone to stop bothering you because you are busy and need to complete your schoolwork. And that it is annoying that they keep bothering you and it is to the point that it is irritating.

They are pretty similar in meaning. I would say they all can be used to mean something is annoying you. However, bothering can also be used to mean that something is concerning you. For example: "The fact that my heart keeps skipping a beat is starting to bother me." It's not annoying, but you are concerned about it.