Hello. :)
Actually, my first thought was, "Yes, it's rude."
Well,...let me explain, because it really depends:
It depends WHO you're talking with and which CONTEXT you're in AND if what you did was for YOURSELF or for someone else.
Examples:
If you're talking with your family/friends/co-workers about wasting your time doing something (FOR YOURSELF), then it's definitely NOT rude, because you're only talking about yourself.
E.g. "I wasted my time day-dreaming when I could've done my homework!"
If you're talking with your co-workers/boss/supervisor about wasting your time just standing there (and waiting for customers to come eat at your restaurant), then I would consider it rude and inappropriate to talk about, because you're doing something for your boss, not yourself.-->you're suppose to take care of customers even if there're few to none.
E.g. "Boss, can I do something else, because I'm just wasting my time standing here. No customer is coming to eat at this time." This sounds very rude. It's as if you don't want to work there or your time at work is not worth it.
If you're talking with your family members about work, then it may be okay--it's still slightly rude, but since the listener is not your boss/co-worker, then it's not as rude...
E.g. "My boss just told me to stand at the cash register even if there are no customers. I felt like I wasted a lot of time just standing there." --> This is still a negative/rude sentence, but your boss will not hear about it, so it is rude but also not rude..Haha, because you're still saying that your time at work could've been spent on something more worthwhile.