It has two very different meanings. The first and more common use would be say it to someone who is in a difficult situation, and lets them know that you are aware. It would be an understatement showing sympathy. But you could also say it literally. Sister A: You're just envious of me because I'm married and you're still single. Sister B: I don't envy you at all. I think your husband is a complete loser. I feel sorry for you. (A thought there was envy. B refuted it.)

It is a set phrase. It's a way of expressing sympathy and not rude. It's used to express sympathy for minor difficulties. e.g. - I have to get up at 5am tomorrow. - I don't envy you! Don't use it for major problems - redundancy, bereavement, etc. It's quite jokey and not strong enough.

Not rude. You might say it to someone who is in a difficult situation.