This happens in English - it is intended to provide some extra information about the journey you will have to take to reach the destination but it is often used inaccurately or with little thought. But yes, it is likely that the person visiting the bay either needs to go south or descend down slightly to visit the bay (which would make sense in a town built around, but probably slightly higher than, a bay or one where the bay is at the south of the town). We would also say things like: I’m going over to visit my friend The car park is round by the shops. I’m going in to town He lives down by the beach She lives up by the windmill

This use of down is strange in English. It essentially has nothing to do with normal definitions of down. It basically means the same as 'over by the bay.' That is, it might be down, but it might not. People will use the phrase, go down there, etc. even when there is no descent involved.

It could mean south. It could mean that the bay is downhill from where the speaker is.