Great question actually. It's related to another expression in English: "to go ape" which means to 'go crazy' in the sense of acting 'crazy' like monkeys we see at the zoo act, at least in the old days when they were behind bars, or how they were depicted in old Tarzan movies. Screeching, jumping around and being wildly excited or agitated. So, it's not a big stretch to then see how "driving someone (to eat) bananas", essentially means the same as "going ape". Drive you bananas = turn you into a hoping, wild banana eater (i.e. a monkey).