#3 is correct.
You've used the word "whom" correctly in #1 and #2. Educated English speakers, who were speaking careful and precisely, would say "whom." However, in casual conversation it is very common to use "who" in place of "whom," and the error might not even be noticed.
"won the prize of excellent young scientists" is wrong and puzzling. It is a little bit hard to know how to fix it. Since a "competition" often has a "prize," you could just say
"The students whom you had recommended to join the competition won the prize."
The preposition needs to be "for," not "of." It would be grammatically correct but unnatural to say "...won the prize for excellent young scientists."
It would be more natural to use the actual name of the competition before the word "competition," or, if the prize has a name of its own, before the word "prize."
"The students whom you had recommended to join the Young Scientists Competition won the prize."
"The students whom you had recommended to join the competition won the European Young Researcher Award."