Alex and Jmat have answered, so all I'll do is add something to it. If I am in a town which I know well and I am talking to someone in that town who also knows it well,,, I'd say "Did you go to City Hall?" It's its name. It's the place we both know. If I am in a strange town talking to a friend who doesn't know the town well either, I'd probably say, "Did you go to the city hall?" I'm not using it as a 'name' really. More a place where my friend needed to go.

It should be "the City Hall", but frequently in English, we do tend to to miss out "the". So "Did you go to City Hall?" and "Did you go to the City Hall?" are more or less interchangeable in speech, but the latter is more correct when writing.

You're misremembering the rule. In general, you should never put an article in front of a proper noun. It make sense logically: a proper noun is the name of a specific instance of something, so an article is completely redundant. Many proper nouns do have an article though, mainly those derived from common nouns. Technically an article is never in FRONT of a proper noun, instead it's PART of the proper noun. We don't capitalise articles or prepositions in a proper noun as convention, but "the" is as much as part of the name of "the People's Republic of China" as all the other words (although it in itself doesn't convey as much information ad all the other words). It sounds more natural to include "the" in the name because "republic" is a common noun, but you would never say anything like "the China". "Charles Street" is just the name of the street, so you don't say "the". It appears that in this city the public gardens were named "the Public Gardens" and the city hall was named "City Hall" because someone completely lacked any imagination. I would expect "the public gardens" to just be a common noun with an official name as something like "Smithville Public Gardens".