If the street is on a hill, then "up" and "down" will refer literally to going up and down in altitude. But probably this is not very often.
In my experience, people VERY OFTEN say "go down the street" or "go up the street" when the street is flat. The up/down is just personal preference of the speaker, regional differences, local culture. (I have often wondered exactly this question: why do some people say "up" and others say "down", for the very same street.)
More examples where "up" and "down" is said, but really has no meaning:
"going down the pub" (a very British thing, and never "up")
"going up to London" / "going down to London"
"going up to town" / "going down to town"
"going up to the shops" / "going down to the shops"
"going up to see my sister" / "going down to see my sister"
Sometimes up/down refers to north/south on a map, e.g. up to Scotland, down to Brighton. However, you could equally hear someone say "let's go up to Brighton for the weekend"
For "come up" and "come down", often the context would imply some physical up/down movement
"come up to my flat on the 10th floor"
"come down to the basement"
"come up the ladder and onto the roof"
"come down from your bedroom [upstairs] and apologise immediately"
However, there are also occasions when - again - it has no sensible meaning:
"you must come up and visit us!" / "you must come down to visit us!"
The use of "come" implies that I want you to travel towards me.
e.g. "come down the pub!" (I am in the pub, please join me)
e.g. "let's go down the pub!" (we are both at home, we will both go to the pub together)
Please note: I am British. I cannot speak for Americans.