The person saying "the conference room is down there" would be indicating, or pointing, in the direction of "down there". It means some distance from where you are standing (as you said, further down the hall, etc.). It doesn't mean downstairs. If the conference room were downstairs, the person giving the directions would indicate so by using "downstairs", not "down there".
Have a nice day,