It all depends on the situation, and where you are located in respect to the other person. The 'down' can refer to physically 'down' - e.g. if you lived in the mountains, and she lived near the sea. It is also often used in a north/south sense - you come 'down south' from 'up north', so in that case, she would live further south than you.
So - in general, yes, you can omit the 'down'. Putting it in there does add more information, even if it may not be clear without more context exactly what the 'down' means.