"over" here means at one's place. They invite people to eat at their place.
The usage of "but" here is not that adequate, since what comes after it doesn't really contrast with the previous statement" we often have people over".
The first statement should include "few people" .
There is also no need to repeat words like " people , over" twice.
A sentence like the following would sound better and more consistent:
" We often invite some ( few) people over to eat, but sometimes we invite a lot of them.