A good question! You would think that as the expression is to go swimming, I haven't gone swimming would be fine - and indeed, for some it is.
However, 'gone' has the idea of not returning yet.
Where is he? He's gone fishing. He is not here because he is fishing.
So, for me, if the question is about now, it's ok.
Has he gone swimming?
No, he hasn't gone swimming he's gone fishing.
But if you are talking about the past, I prefer 'been'.