Okay. In fact, '내려가다' is not a nuetralized expression.
There are distinctively different two expressions.
They are '내려가다'(go down) and '내려오다'(come down)
In English, we say 'come'(오다) when we talk to someone who is far from us.
But in Korean, we have to say 'go'(가다) when we talk to someone who is far from us.
Do you know what the difference is?
English people focus [where the listener is], but Korean people focus [where I am].
'오다' and '가다' is totally different, that's why I told you there is no nuetralized word.
To make easier, '오다' is 'moving towards where I am' / '가다' is 'moving towards somewhere else'
If this is not enough to understand, let me know. Cheer up!