Many of my close friends at university were physicists. Not mathematicians but pretty close. And they were all lunatics. The mathematicians I knew were all pretty outgoing as well. So no, this is not my experience at all. I am reminded of a joke though, related to this subject. 

How do you know when you've met an outgoing mathematician?

They stare at your shoes instead of their own.

In the case of mathematics--university-level mathematics, not accounting or arithmetic--I think we can guess at an association with personality. It is not so much that mathematicians are cold or dry, it's that they are intensely focussed inward.

I actually had a discussion with this once with someone, and dragged out a college yearbook from a year that wasn't ours (so consisted only of strangers). We went through the pictures trying to guess who was a math major, just from the picture. There might have been pictures of 400 students, of whom perhaps 20 were math majors. I picked out about ten people as "looking like math majors." So I missed about ten. Of the ten I picked, about eight actually were math majors, and one was a physicist. Yes, there is a "look" to them.

Maybe our personalities effected by what we study and learn however we can't generalize

I still remember most not all of my math teachers were strict ,coldfish and harsh because of them I hate math and I was very weak at math