We usually use that in terms of romantic relationships. If you say, "He/She is out of my league," it means that you think that person is too good looking, intelligent, or any other good quality to be with you. There's a similar phrase, "Out of my depths," for more generic situations. If a particular subject is too difficult for you to understand or out of your control, you can say it is out of your depths. Example: "Wow, she is so beautiful. She is completely out of my league (because she wouldn't think I'm handsome enough)." "I just can't understand quantum physics. It's out of my depths."

It's a metaphor derived from baseball leagues. Professional baseball players play in leagues. The minor leagues would include players who could be scouted and moved up to play in "the majors" or "the big leagues." If you did not have the competencies to play in the major leagues then you would be "out of your league" if you tried. It thus became a metaphor for competencies; for example, in jobs, musical bands, educational institutions and yes even good looks for some.