"Would you like to go swimming?" is the correct choice. Anytime we say "go to" it implies a direction to a location. For example: 1. Every Sunday, I go to church. 2. We go to the movie theater on the weekends. "Go to swim," therefore, does not make sense. Unless, of course, "Swim" is the name of a city/place. Then, I suppose, you could say, "Let's go to Swim!" :)