"The part that hurts the most" typically refers to the specific thing regarding a situation which causes a person severe pain, although it can also refer to what is hurting. I think in these lyrics, both ways are used. He implies that both people are hurting (the what), but what causes him the most pain about their situation is everything he does without the other person (the why).