домОй and дОма _aren't_ (!!!) case forms of 'дом'! It's adverbs, the fist means 'to home' the latter 'at home'. Домой comes from ancient Russian dative 'домови' (modern Russian dative is дому. What is happened is that there were several declination classes in Old Russian. Then case endings mgrated from one class into another, the whole system undergone changes and -ови become outdated. But in this use (with the verbs of movement) it survived, thus giving birth to an adverb. Nobody recognizes it as a dative form now:)).

дом ( house)- it's noun, you are right and it's masculine вернуться домой it's adverb and we say: Я вернулась домой. Я сегодня не вернусь домой. Когда ты вернешься домой? and you can say : Я уже дома!! or Я вернулась домой!! They are both right and good. Good luck!

"домой" is adverb.

Домой - это наречие. У него нет падежей.