The word "down" has many meanings. The most common refers to a change in height or altitude, the opposite of "up". But one of the others refers to going further along something or further away from the starting point, in any direction. When you use it with this meaning, it is a preposition and usually is followed by an object. "Go down the street." "Walk down three blocks." But when you get to the elevator .... "Do I go up or down?" Here "down" is an adverb modifying the verb "go" asking about which vertical direction.

Good question Diego, we are on the same page.