Климентий,
I had been you.
This sentence is not passive. It is the past perfect active of the verb "to be".
If I had been you = Если бы я был Вами.
I am you. Я явлаюсь вами.
I was you.
I have been you.
I had been you.
I had you. У меня был ты. --или-- Я имел тебя.
I had had you. К тому времени я уже имел тебя. Речь идёт о половых связях