"I prefer" can be followed by a noun, an infinitive (with "to") or a gerund. "I'd rather" can only be followed by a base verb (without "to.")