It's a quote from "Hamlet" by Shakespeare: "Though this be madness, yet there is method in 't" (Polonius) Polonius sees that Hamlet's behaviour seems erratic and insane, but there is clearly a plan behind it. We've changed this quote to "a method to your madness" to mean that there is a plan somewhere, even if it's not obvious.

To add to Peachey's answer, it's become a stock phrase. Many people use it without being conscious of it as a quotation from Shakespeare. Show a native English speaker "there's a method in my __________" and most will say "madness" instantly, without even thinking. The most common use is a lighthearted way of saying "this looks crazy, but there's actually a reason for it." "Here's my way of converting Fahrenheit to Celsius. I start by adding 40. Yes, 40, not 32. I know it seems crazy, but there's a method to my madness..." However, the phrase is so common that people like to use it as a joking way of talking about _anything_ concerning either methods or madness.

Just to add (in case it wasn't clear) Your psychology professor was making a joke. As the answers below said, the phrase "there's a method to [my] madness" usually means that while your actions may seem kind of crazy, you really do have a plan that makes sense. But when he used it, he was making a play on words. Psychology is somewhat associated with "madness." So, a method for psychological experimentation is a "method to madness." Ha ha. It isn't a very good joke.

Usually it means that even though somebody is doing something that seems crazy, maybe they actually have a plan and a reason for what they are doing.

Maybe a dialect difference here, but I have frequently heard and used 'method in my madness', meaning exactly the same thing as 'method to madness'