There could also be differences in meaning.
I prefer swimming to sailing. (You are comparing two activities in some unspecified way. For example, You might prefer to watch swimming on TV)
I’d rather swim than sail. (You are talking about activities that you do.)