I am not a native speaker of English, but according to my knowledge, "four of you" can be different depends on the context. Maybe the number of "you" is only four. But at the same time the number of "you" is more than four. It may mean four out of 100. My answer to this question is that the phrase can differ according to the context.