The dictionary will show whether a verb can be followed an infinitive (to "verb") or a gerund ("verb"-ing). The verb "afford" takes an infinitive.
Collins online dictionary
We couldn't afford to buy a new rug. [VERB to-infinitive]
New Oxford American Dictionary
(can/could afford) have enough money to pay for:
[with infinitive] : we could never have afforded to heat the place.