"Carry on" is not rude if used in a general sense of indicating that someone continues with a task or activity. As with any language, much depends upon context, attitude and tone of voice. "Carry on" can be rude if said sarcastically and with intention to be rude.

It's certainly not inherently rude, but just about anything can come across as rude depending on the circumstances and on your tone.