Great question actually. It's related to another expression in English: "to go ape" which means to 'go crazy' in the sense of acting 'crazy' like monkeys we see at the zoo act, at least in the old days when they were behind bars, or how they were depicted in old Tarzan movies. Screeching, jumping around and being wildly excited or agitated. So, it's not a big stretch to then see how "driving someone (to eat) bananas", essentially means the same as "going ape". Drive you bananas = turn you into a hoping, wild banana eater (i.e. a monkey).

Imagine the behavior of a lovely, hungry and excited monkey if you put a banana in front of him... =D (so cute) ...ok, That's going bananas ^^ I think it's the same as "go nuts" . ^^

Other synonyms in British English are to "go mental", to "flip out", or to "go off it" (although they can all mean both 'go mad' and 'get angry' depending on the context).

We like bananas.

drive me bananas = crazy. It means you're driving me crazy. Same as 'you're driving me bonkers.'