Yes, it's either "would" or "had". You'll know which one it is based on the following verb. "Would" is always followed by the verb in the base form, and "had" (ie. past perfect "had") is always followed by the past participle.
You'd ask = you would ask
You'd asked = you had asked