It's not being used as a phrasal verb like when you "try on" clothing. Here "trying" means "difficult" or "challenging" and "on you" or "on me" tells us to whom it was difficult.

"trying" means difficult and annoying. It is followed by a preposition "on + somebody". I had a very trying day at work. (it was very trying on me) Julie’s having a trying time at work lately. (it was trying on her)