Where is correct. Learned seems a strange word here. Some of them might not have learned it. More normal is “the participants were set a task…”

Only the first one is correct. In the second one, the relative pronoun "that" is replacing "task" in the dependent clause. "That" is the subject of the clause. When a relative pronoun is the subject of a clause, you can't follow it with a subject pronoun. You could say "...a task that required them to recognize..." You could also just divide it into two sentences. For the study, participants learned a task. They had to recognize a hidden pattern in the questions they were asked.