English grammar is descriptive, not prescriptive, and as English is rather irregular, it can certainly be confusing. Firstly - " I have to go see the doctor." isn't general English. To me, that's rather American sounding. I would always say "I have to go to see the doctor" or "I have to go and see the doctor." Secondly, are you sure it's "Here's a..."? that sounds rather odd. Are you sure it's not "There's a...". And thirdly, "I go to swim every day." isn't completely incorrect. It's certainly less odd than "Here's a... pool". If you made it "I go there to swim every day." it is perfect. It's just that the gerund form is more common in that case.

Sometimes a verb with -ing ending serves as a noun. It is called a gerund. the verb "go"+ a gerund only fits with some verbs. I would memorize a few phrases. I go running. I run everyday. I go walking at the park. I go sailing in my boat every weekend. I + go to + ___________ (a location, a specific place) I go to the doctor. I go to the beach. I go to my grandmother's house.