In 뭐 하는거야, it uses "Verb+는 것" structure in it which is the way to make a noun form out of Adjective/Verb. We use it to try to get confirmed in questions. So that structure with 뭐 which is What is emphasizing more the speaker's emotion which can be really curious about or being disappointed or whatever the speaker has in his/her mind. 뭐해 is just a simpe question about "what do you do?" with no special feelings or intentions behind. Hope this can help you!

i think there's no much difference. But personally, I normally say '뭐해? ' just for asking casually, But when i feel something is going wrong or when i snap at, i say '뭐하는거야?'.

'뭐 해?' is simply meaning "what are you doing?". But with the stressed sound at the first word like '뭐↗ 해?', it means "what the heck are you doing?". The same goes for '뭐하는 거야?'. But I think this is usually used with some tones more than '뭐 해?'. I guess it's because that it has some more words you can make it so that you could give some tense changes or what not. Such as... "뭐↗ 한 거야?" = What the heck did you do?