It means they're becoming stupid. I don't know if this specific saying is actually used in the Southern US, but there are a lot of sayings like it.

"10-gallon hat" is a cowboy-style hat that the top is exaggeratedly big. In this context, I'd say the sentence means that some of the people the "other brainiac" invited/recruited are leaving or have already left.