I've never seen this television show before, but I guessed correctly that this character is a Southerner. This style of speaking is somewhat cultural. I also live in the South. This may or not be rude. It does sounds a little forward though. Yes. But if you wanted to change it, then it would be more polite to also ask for the seat. We would say something such as "Do you mind?", "May I have it back?", or something similar to that. But even these could be rude in some contexts and depending on the tone of voice and body language. But the most grammatically direct way to say this would be to say close to this: "I think that you are sitting in my chair. I was here before (or add whatever details that you have to explain why it is your seat). May I sit here?"