It's like if you are discussing with your friend about how important it is for a supermarket to be close to your home. Your friend says it's not that important, and you say "You have a car", because you don't have one, so you have to walk or take the bus. Now, is it an insult? Is it a factual statement? Does it annoy your friend because it points out their privilege? If yes, he will start talking about how old the car is, or how it's not a Ferrari so it doesn't count.

Thank you Tanya!

Magnificent answer, Tanya!

This really depends on the context. I assume that this statement refers to the color of someone's skin in reference to race, so my following comments will relate to that. Sometimes people can be teasing when they say this. Or it can be a simple statement. But sometimes, people will use this phrase as an insult, to "other" or put someone outside their group, in a bad way. Without delving into race relations (as I know them in the United States), I will say that this phrase can have many meanings depending on who says it, how, why, and when. I would be careful using it with Americans, since we have a complex system of race politics and you never know what someone will think of your words. Although I can only speak for America, I know other English-speaking countries have difficult race relations at times in their own lands. It's a topic to study up on, for sure. Like in your native language, try to pay attention to the intent of the person using the words before you respond, or try to learn about the person before you say something that could be controversial.