뭐하는 거야 = short for 뭐하는 것이야 (or 뭐하고 있는 것이야 in this case) 것 is a pronoun, used like 이것, 저것, 그것. 공부하는 거야 is same with 공부한다 + 것이야. You can guess 것이야 is just to emphasize 공부한다. In most cases, 거야 can be omitted or replaced by 것이야 and has same meaning. 거야 emphasizes the sentence or put some subtle meaning in it, which I cannot explain clearly. Another rule for 거야 is like this. One ask, 오늘 뭐 할거야? You reply 집에 갈거야. (집에 갈 것이야.) 오늘 뭐해? 집에 가. has same meaning. But 집에 갈거야 has future tense (가+ㄹ). For this future tense, 것이야 has to be added. It is too bad korean has complicated and unclear (at least to me) rules for verbs. Same meaning, lots of expressions, and all have subtle differences which are hard to explain.

Well, I'm not a native speaker, but I'll give it a go! Did you mean "뭐하는 거야?" That means "What are you doing?" Firstly, '거야' is in the informal speech, so you probably shouldn't use this with someone older or of a higher title than you. Secondly, in this context of "뭐하는 거야?", the 는 거야 is used to indicate present tense. For example, "뭐하는 거야?"- What are you doing? "공부하는 거야."- I'm studying. You can also use 거야 when referring to the future tense. However, you must conjugate the root verb. For example: 하다- to do. " 뭐 할 거야?"- What are you going to do? If there is no consonant ending the word, you use 'ㄹ 거야' If the word DOES end in a consonant, you use "을 거야." (for example: 먹을 거야) The formal form of '거야." is "거예요." For example: "선생님, 오늘 뭐 할 거예요?"- Teacher, what are we going to do? I hope I helped you! If you're not satisfied with my answer, then message me and I will ask one of my native speaker friends to help you!