"If we invited Ben (to the party), we'd have to invite his friends too." --This sentence is wrong. It changes tense after the comma. Corrected versions: If we invite Ben, we have to invite his friends too. If we invited Ben, we'd have had to invite his friends too.
'would have' is a modal auxiliary. The main verb is 'invite' in this clause. 'Invite' needs to match tense in both halves of the sentence.
This is my understanding. I hope it is helpful.