Let's look at some simpler sentences:
"They sang." (No object or indirect object.)
"They sang a song." ("Song" is the direct object.)
"They sang Bruce a song." ("Bruce" is the indirect object.)
Now let's do the same with "ask":
"They asked." (No object or indirect object.)
"They asked a question." ("question" is the direct object.)
"They asked Bruce a question." ("Bruce" is the indirect object.)
Now let's change "a question" to "to learn the answer":
"They asked to learn the answer." ("to learn the answer" is adverbial.)
"They asked Bruce to learn the answer." ("Bruce" is the indirect object.)
Now, let's get closer to your sentence:
"They asked for a speech." ("for a speech" is adverbial)
"They asked Bruce for a speech." ("Bruce" is the indirect object.)
Finally, let's replace "for a speech" with "to make a speech". Grammatically, these phrases play the exact same role:
"They asked to make a speech." ("to make a speech" is adverbial)
"They asked Bruce to make a speech." ("Bruce" is the indirect object.)