We just don't usually use who or whom in this sentence. Just say - He's the man I saw yesterday. In practice, we seldom use whom these days. There is a difference in the grammatical meaning, but it is becoming old English. Take a look here: https://en.oxforddictionaries.com/usage/who-or-whom