We can use this to talk about emotions. You are saying you won't stop feeling something. I'll never let go of my love for my girlfriend. I'll never let go of my anger for my girlfriend. You could also use it in a more physical sense if you are holding something. You can say The dog won't let go of the ball.

I imagine a woman. When I say: "I will never let YOU go" it means that I wont let her to go away and leave me alone, because I need her and I love her.

Maybe it's mean "I don't give you leave me" or something like that.)