It's very specific, so I doubt you can use it. He is saying he is 'vulnerable' - i.e. that he can get emotionally hurt or upset easily. Intensely vulnerable means he has a particularly strong reaction. 'don't let something be a factor' here means 'don't let something influence your decision, or judgement'. And the implication is that since he is vulnerable, a rejection might really hurt him. So, he is asking her not to worry about hurting his feelings, when she thinks about her reply to him asking her out. So - unless you are 'intensely vulnerable', then a direct use of it probably isn't useful. You could possibly modify it - you might get an interesting reaction if you said 'try not to let my intense violence become any kind of a factor here...'