Choosing between the TO + Infinitive and the gerund is easy if you remember the simple underlying rule.
* To + infinitive => concrete action here and now; what one intends or desires now or in near future; subjective sense.
* Verb + -ing => one's general routine or preference; something one has been doing; objective sense (more factual).
Examples:
1. "prefer"
-- A. If I like going fishing more than hiking => I prefer going fishing (to ...) - general fact based on past experience.
-- B. If I want to go fishing this weekend => I prefer to go fishing this weekend - concrete intention for near future.
(A) can go with the infinitive too, but (B) is definitely better with the infinitive.
2. General preferences can go either way because liking something in general means you like it now too.
- I like going fishing == I like to go fishing.
- I love watching old movies == I love to watch old movies.
3. But "will" and "would" mean concrete intention or desire, so the infinitive should be used.
-- A. I would love to go out with you => I am excited to go out with you now or in the near future.
-- B. I love going out with you => I liked going out with you in the past and still do.
-- C. I would love going out with you => BAD - "would" is for concrete desire, so won't work with a gerund.
4. Some verbs can take either, but their meaning changes because of the above the rule.
- I stopped smoking => I quit the habit of smoking which I have been doing until now.
- I stopped to smoke => I stopped (what I was doing) in order to do something now, which is smoking.
- I remembered seeing her => I recalled something I did in the past, which was seeing her.
- I remembered to see her => I recalled that I had something to do, which was to see her.