We usually use that in terms of romantic relationships. If you say, "He/She is out of my league," it means that you think that person is too good looking, intelligent, or any other good quality to be with you.
There's a similar phrase, "Out of my depths," for more generic situations. If a particular subject is too difficult for you to understand or out of your control, you can say it is out of your depths.
"Wow, she is so beautiful. She is completely out of my league (because she wouldn't think I'm handsome enough)."
"I just can't understand quantum physics. It's out of my depths."