I'm afraid that I have to disagree with James. The tag question verb has to echo the auxiliary or modal verb in the main statement. In this sentence, 'had' is not an auxiliary verb - it's a main verb in the past simple. The underlying auxiliary verb in simple tenses is always a form of 'do'. Here's a tip - if ever you're not sure what verb to use in the tag question, put the statement into a direct question or a negative. In this case, the question is 'Did he have to buy bread?' and the negative is 'He didn't have to buy bread.' This tells you that the underlying auxiliary verb is 'did', so the full tag question is 'He had to buy bread, didn't he?' The tag 'hadn't he?' is used in sentences where 'had' is the auxiliary verb, for example in the past perfect, 'He'd already bought bread, hadn't he?'.

Keep the verb consistent: "He had to, hadn't he?", "The butler did it, didn't he?", "We should go, shouldn't we?", "You'll come, won't you?", etc.

To detect the tag verb, make it a question or a negative: I didn't have to buy bread - this auxiliary verb will always reappear in tags.