The price to pay emotionally would be too high.

2 things, either you love someone and you don't want them to go because it would hurt too much, or someone you work for is telling you that you're too important. To afford something, is usually connected to money, yes, but it can also mean give, "I afforded her the chance to explain". I can't afford to lose you, tends, like I said, to either be about love, or importance for what you do for a company. Emotional pain, or financial/operational detriment.