I had to go! Had is already in the past. You cant have two past simple verbs following each other like that. Good question though!

I had to go. This is correct because the main verb in the sentence is “had” which is already in the past. “To go”, therefore, is being used in the infinitive form, which does not get conjugated into the past tense.

I had to go