If French is your native language then think of who/whom as qui/que (for people- for things we use 'that/which')

Pyotr, l'homme qui m'a vu hier... -Pyotr, the man who saw me yesterday

Pyotr, l'homme que j'ai vu hier... -Pyotr, the man whom I saw yesterday

So far so good. But I think that your confusion probably lies in the fact that most people do not use the who/whom distinction: so you have probably seen 'The man who I saw'. That is so common now that it is almost standard, although technically it is incorrect. 'Whom' for object is becoming very old-fashioned sounding and very few people use it at all. 

In sum, the rule itself is easy for you to understand as a French native, but you will get confused if you expect people to use it properly, because most do not.