The difference is geography: north is "up", south is "down". You can say: "I have to go there..."

"Up" should indicate that the bank to go to is spatially higher than where [The talker] is currently located. Given that, they can be used interchangeably.

The 'up' and 'down' can mean a few different things in different contexts. "I have to go down there, and deal with them (the bank)" - this probably refers to going downstairs, unless the bank is a long way away. If it is referring to going downstairs, down should generally be included in the sentence. "I have never been to LA, but I am planing to go up there next spring" When referring to long distances, up or down refers to going north or south. If you are going north, you can say you are "going up there" or "going up to LA". When talking about long distances you can drop the up and down from these sentences, but native speakers will often include them.