I don't really know how to explain specific grammar rules that well but I'd say it's because there's the present perfect 'have' in the sentence.
If it were just 'would' and not 'would've (would have)', it would be 'Would much rather see him', but because it's 'would've' then the past tense, 'seen', is used :)
e.g. I have gone there --> l would have gone there
I have bought that --> I would have bought that
I have seen him --> I would have seen him
etc etc
Hope this made sense and if it did, hope it helped!