Just like Marko has just explained to you.. we use dom for home, so it means that we really belong there, it is not just a simply physical place.
However, if you see a house just like a phyisical place and you have no feelings about it, I mean..you do not feel like you belong there, then it is going to be simply "kuća".
For example: Ja ovdje živim, to je moj dom. - I live him, this is my home.
Ova kuća je lijepa, ali nije moj dom. - This house is nice, but it is not my home.
Do you get it?
If you have any further questions, feel free to ask me.