This is not any general English slang but perhaps more regional or personal, because it covers up meaning in the song. Although with this particular lyric from Nicki Minaj, it is believed that the bananas symbolises the f*cks that she gives and that the primates or chimps are her hates; so she is not going to care about her haters.

i don't know