Yes, it's subjunctive mood. By singing "if I were a boy" Beyoncé is admitting that she is not, and was not ever, a boy. She simply imagines it.
If she sang "if I was a boy", she'd be treating the idea as a strong possibility or even reality. Maybe she can't remember her childhood and wonders if she was given a sex change at some point.
You'll find speakers using "was" in place of "were" when imagining weak possibilities. This is considered a bit careless, so pay attention and stick with "were".