If I had been asked...I would have answered. It is correct in terms of the conditional, but you have to keep the subject of the sentence constant. Yes you can use the passive with the conditional... It is fairly common, especially when we cannot specify who was doing the asking. If I had been bitten by that lion, I would never have come to England. (Taking of a specific situation that the heater knows). If I had been born a male, I would have had greater privileges in life.

I very common example: "I studied the wrong subject when I was in college. If I had known then what what I know now, I would have studied computer science."