You've misquoted (or, you're using a shoddy paraphrase).
"It would be better if there were but one inhabitant to a square mile, as where I live. The value of a man is not in his skin, that we should touch him." - Thoreau (Walden)
It means a person's value is not found in the physical form, if we wanted to make contact with that value.