A appears to have doubts about her boyfriend, because he’s not in school or not doing well in school. B brings up the fact that A’s boyfriend had an affair with a teacher. That’s not helping. (What you said doesn’t help me feel better about my boyfriend. It doesn’t help me have more confidence in him as a boyfriend. It doesn’t help my doubts go away. It’s a set phrase)

You are correct. A means that B is not being helpful. I imagine this scenario as A having a slight sarcastic tone when she says "not helping" because 1. B is literally not being helpful; and 2. B is bringing up the subject of Sebastian's affair with the teacher that A would probably rather forget. B would probably say something like "just trying to joke with you" or "lighten up, I'm just joking" as she probably said "not too cool for school teacher" with an emphasis on the word "teacher" and maybe with a little smirk or smile on her face. Hope this helps.