I was confused till I read the comments because I had never heard anyone say this in Chinese. In the US, we, the Asian Americans, are at times called "bananas". It could be used as a pejorative term or as a non-pejorative term.
We do tease each other (Asians) with terms like that, such as whitewashed, banana, fob, etc. But I suggest you avoid using any of those slurs unless you are talking to a really close friend.