Yes, it is natural. Sometimes, however, people use that phrase when they ARE being rude. Example: "Miss, I don't want to seem rude, but your dress is really ugly." But it can also be used politely. Example: "I don't want to seem rude to the teacher, so I won't criticize her even though her lecture was boring"