My dictionary seems to agree with you. Here is its example: "[I am] going DOWN to the office." It defines "down" in such an sentence as an adverb that indicates TOWARD a center of activity. (Source: The American Heritage Dictionary of the English Language.) ***** Another dictionary has this sentence: "I'm going DOWN to the arcade." It says that this use of adverbial "down" indicates movement to a place that is perceived as lower. (Source: The New Oxford American Dictionary) ***** Finally, Webster's Ninth New Collegiate Dictionary defines it as an adverb that means "to a point away from the speaker."

We'd use to say the phrases 'uptown' or 'downtown' when stating a direction we were going depending on what part of town we lived in. Recently English speakers have dropped the 'up' all together and usually state everything as being 'down' even if you our literally going up (North) to a place. Using 'down' in this way is a type of slang that replaces the use of other prepositions when going somewhere. Down to the corner, down to my house, down at the store, down to Canada (Maybe no one says this one though.)

Hello. I am not a native speaker but I think I can give you an answer. Words like 'on', 'in', 'up', 'down' (prepositions and postpositions) are always confusing, and there mostly is not a general rule to decide what to use. You just have to read and listen to a whole lot of English to use them in the right way. 'Down' is not really indicating anything I fear. It's just used in the expression by custom. Even for closely related languages, other prepositions are used for similar sentences: English: I am in love with you. German: Ich bin in dich verliebt. But Dutch then makes the exception...: Dutch: Ik ben verliefd op je. ('op' in Dutch equals 'on' in English) To make it worse, sometimes it is even depending on the dialect within a certain language which preposition you use.