It all depends on the context, and on what has been said before. The fact that you say THE before department indicates that the listener knows which department you are talking about. If you are showing a visitor round your company, you might say "Welcome to Marketing. There are six of us in the department." In this case, it's clear that you are saying that the department consists of six employees. There is no ambiguity. Without further context, there is no suggestion that the department is any larger than six. Six means six - not nine. Your second interpretation is not possible. For this to be the case, you'd have to say something like this: "Some people are going to lose their jobs. From next month there'll only be six of us in the department."

I can't see any confusion. Saying something about the size of the department says nothing about the size of the company. By saying 'department' you are _probably_ suggesting there are more people in the rest of the company, but you would not make any assumption based on that. Your use of 'men' is confusing though. You said 'us'. You didn't suggest any gender.