'At' can be for any location, whether it's a building or not. For example 'at the bus stop' .
'At' also often refers to the function of a location rather than the physicality of being inside a building. Think of 'at school' or 'at work' - these uses of 'at' indicate that the person in question is studying or working. The focus is on why they are there.
So, if you say 'He's at the pub', this could mean that he is inside the building, or it could equally well mean that he's outside sitting at a table in the pub garden, or on the balcony, terrace, or even in the car park. It's a far more general location. We know that the person is probably relaxing and drinking with his friends at particular location, but we don't know whether he is indoors or outdoors.
If you say 'He's in the pub', this means that the person is literally inside the building. So you might say 'Fortunately, I was in the pub when the thunderstorm struck'. You'd use 'in' here because you want to indicate that you were physically indoors.
The same goes for 'house'. A party in your house is indoors. A party at your house could include your garden/yard/patio etc. In your example, we say 'at my house' because you are referring to the function of your house - it's your home and you are offering your hospitality there. The reference to the rooms, by contrast, refers to the architecture of the house and the fact that the rooms are physically located inside the building.
I hope that all makes sense.