"Being on your case" is when someone is pestering or bothering you about something that you might have or have not done. You can imagine yourself being a suspect in a crime and the "police" (can be anyone: your friends, your parents, even actual police) have a "case" on you. Telling them to "get off your case" means that you're feeling uncomfortable with the scrutiny and you want them to stop mentioning the thing you might have or have not done.

Depends on who you are talking to. If you are talking to the person who is bothering you, "OK, OK, just get off my case!" If you are talking to else, "He is always on my case." Hope that helps.