It means someone did something extraordinary or something extraordinary happened, probably in its negative meaning, and you ask that person or just anyone why did it happen. Sometimes you don't even need the answer and you say this phrase just to express your attitude to the situation.

For example you eat your lunch and your friend comes and eats your cookie, you say "what the hell are you doing, man?!", because you were expecting to eat that cookie yourself :)

They're probably saying "What the hell, man?". See http://www.youtube.com/watch?v=sQpUoxyMjj4