Hi Olga,
And yes, you are correct. If he has already done it, and you wish to ask whether it were his first time or he had done it before that, you would use "had".
Note: Instead of saying "gone", I would say "been"; so, "had you been there before, or was it your first time?"
And "have" you would use if you are asking about before the present. So, "have you been there before?" you would aks if your friend is yet to go, and you wish to know whether he has been there in the past.