The difference is very subtle.
When you say “There were just six of us” you are merely announcing how many people there were. Then maybe you can start enumerating those people, (i.e “There were just six of us; me, alexander, sofia, ivan...” and so on). However, the main focus on this type of sentence is on ‘us’ and not on ‘who specifically”.
“US” can be a nationality, a team, a race, or any other unifying attribute that merges certain group of people along with you.
For example:
“When I was little I went abroad to study in Britain, and surprisingly on our school there were six of us.”
“I didn’t know anyone else knew about the new law, but can you believe it? On the bar I went last night there were six of us who knew”.
On the other hand, “There were the six of us” it is implied that the six people are already known by the listener. In other words “the six of us” are individuals and not a group as in the previous sentence.
To put it simpler, by adding “THE” you are referring to something specifically. For example: A car, can be any car; THE car, is a specific materialization of the concept “car”: The car of my friend, the car that can drive 20 miles, the car that talks, etc.
Hope this helps!