It means a remainder of people. For example: Some of us are going to the cinema, the rest of us are going to the park. "The rest of us" means that these are the people not going to the cinema :) Hope that helped!