It could mean that. You can only use "the absent" in a situation where its meaning is understood. Without knowing the situation, I cannot tell you what it means. It could mean many things. For example, perhaps there are some birds that did not return in the spring from their winter migration. Those birds could be said to be "absent". Maybe they flew to France instead of England. Then you could ask a French person "are the absent safe with you?" Strange, but possible.

A professor once put up a sign saying "The absent are safe here" it meant that neither he nor his wife, would speak unkindly of those not present (not talk nor gossip about them behind their backs).. https://www.keepbelieving.com/the-absent-are-safe-here/ .. you can read about it here. Your sentence therefore means, guests and visitors will not be spoken about once they have left the property/building/place .

You are right. Are the people who are not here safe with you? A bit of a strange sentence though.

Did you mean displaced people? "Are the displaced people safe with you?"