So if I understand correctly, you are trying to change the subordinate clause "If you want to learn well in this course" to past tense, right?
First I would like to address that the sentence is a tad awkward in my opinion. Perhaps it would be better to rephrase it "If you want to do well in this course, you should learn more information."
I hope that doesn't change the meaning from what you meant to say.
If you want to change the tense, you need to make sure both clauses are changed.
"If you wanted to do well in this course, you should have learned more information" is how I would write this in past tense.
It would not make sense to only change the subordinate clause:
"if you wanted to do well in this course, you should learn more information" (incorrect)
If you wanted to start the sentence with "he said that", I would write it as such:
(assuming the "you" addressed in the sentence above is me)
He said that if I wanted to do better in this course, I should have learned more information.
(assuming the "you" addressed in the sentence above is a third party)
He said that if he/she wanted to do better in this course, he/she should have learned more information.
Hope this helps :)