I think it depends on which way you are using it. If you are saying "I can't afford this shirt," then I usually say "No puedo pagar por esta camisa." If you are saying something more idiomatic like "I can't afford to take a risk," then permitir would be your best bet. "No puedo permitirme que correr un riesgo."