This is definitely not an easy one to tell exactly why it's different. The important thing is to recognize that they are different in this case. Many times in languages there isn't a clear reason why something is said in a particular way, it just evolved that way to have a meaning that people understand. Languages don't follow rules, the rules are made up to try to describe existing use of language.
In any case, the first sentence starts with never and means there hasn't been a time before that that large number of people have visited. The second means that there are a lot of people that have never visited the city.
I think in this case the difference is that "so many people" is an indefinite amount. When that is combined the different positioning in the sentence with "never", "so many people", and "visited", it leads to the action being different. That's really the only difference in the two sets of sentences, so that must be it. But I can't come up with a specific reason why.