Yes, there are some rules but sometimes you can use both пятеро and пять (like in your examples).
So, with the word which name people (люди, дети, студенты и т.д.) of masculine or general gender you can use both пятеро and пять. But not with feminine.
У него пятеро/пять братьев.
У него пять сестёр.
Other case is when you use substantivized numbers meaning people:
К нему подошли трое.
In this case only collective numbers (трое, пятеро и т.д.) are correct.
And finally you use collective numbers with pluralia tantum word: двое брюк, трое ножниц.
But only in nominative case! In oblique cases you use cardinal numbers: двух брюк, тремя ножницами.
Quite tricky, isn't it? :)