I agree with most of Cole's answer, but I find both his definition and his examples of "catharsis" extremely surprising. Catharsis does refer to something that relieves stress or negative emotions, but not in a pleasant/soothing way. A "catharsis" is, in itself, an intense emotional experience -- it's not likely to happen while you're sewing or reading (unless you have a very strong emotional reaction to the book).
The idea of "catharsis" comes from the Greek philosopher Aristotle, who used the Greek word ("κάθαρσις") to refer to an act of emotional "purging" or "cleansing," specifically associated with watching tragedies. He believed tragedy was a valuable form of art because it allowed people to release their own sorrow and fear by responding emotionally to the terrible things that happened to the characters onstage.
Today, we talk about experiencing "catharsis" by watching emotional films, listening to emotionally intense music, or talking about our own emotional experiences. Catharsis is usually somewhat traumatic while it's happening (often involving shock, fear, or sadness), but in theory, we will feel more calm and happy AFTER the catharsis.
I have not seen Game of Thrones, but I believe that "cathartic payoffs," in this context, means some kind of positive and calming outcome FROM the extreme violence. In other words, we would expect a work of art to use extreme violence to create a cathartic experience for the viewers (the shock that they feel while they watch the show lets them release their emotions, so they feel more calm afterward, in real life). The writer seems to be saying that Game of Thrones has not previously been offering this expected catharsis. There has been a lot of violence, but the violence was meaningless and did not result in any positive "payoff" for the viewers.