As Su Ki mentioned, "up" and "down" can refer to altitude.
We say "down at the beach" or "down at sea level" as Su Ki mentioned. We also say "up in the hills" or "up in the mountains". So, if someone has a cabin in the mountains, they might say "up at the cabin".
With reference to rivers, "up" and "down" also refer to "upstream" and "downstream".
We say "up in the stands", "up in the bleachers" or "up in the nosebleed seats" to refer to the highest and least expensive seats in a stadium.
As Su Ki mentioned, "down" and "up" are not always physical in nature.
One figurative meaning of "up" and "down" comes from New York City. In New York City, "uptown", "midtown" and "downtown" refer to northern Manhattan, central Manhattan and southern Manhattan.
However, in American English at least, "uptown" and "downtown" have entered the language with connotations that transcend these geographical meanings. "uptown" suggests affluence and a more sophisticated and urbane culture. "downtown" suggests more hip and stylish in a more working class way. Billy Joel has a song that talks about the relationship between an "uptown girl" and a "downtown guy".
New Yorkers also refer to "upstate" as being any part of New York State that is not New York City (because NYC is at the southernmost part of New York City). I don't know if any other state uses "upstate" in this way.
"up" and "down" can also take hierarchical meanings. "up on the executive level" is used when the top managers have offices on the top floor of the corporate office building. "up at corporate" suggests that the corporate office building is figuratively higher than the other workers.
At school, "upper classmen" are juniors and seniors (as opposed to freshmen and sophomores).
"down at school" or "down at the hospital" are new to me. Can you provide some context about these usages?