It doesn't have anything to do with being rich or poor. It means they are being hypocritical or that they are criticizing you for doing something that they do themselves. It isn't really insulting exactly, although it can be. It's more about calling someone out on their hypocrisy. For example, if I was known for being very greedy and I accused you of being greedy you might say "that's rich coming from you!" or even just "that's rich!". It's a way of saying I'm being a hypocrite.

It’s an big insult. It really is saying you don’t have the right to talk like that. It infers that you are “poor” and do not deserve to speak like a “rich” person. It is a put down (insult).

Duncan, your explanation was much better than mine!!