What a good question - I had to think about this.
There is a very small difference in meaning between the two forms. The -ing form emphasizes the action or experience. The to-infinitive gives more emphasis to the results of the action or event. We often use the -ing form to suggest enjoyment (or lack of it), and the to-infinitive form to express habits or preferences.
I hope that helps! Best wishes!