One of the meanings of "wear" is "To damage, diminish, erode, or consume by long or hard use, attrition, or exposure. Often used with away, down, or off." So the usage in this sentence is describing something physical (the erosion and breaking down of an object or tool) to describe something that happens to people's bodies, minds, or psyches: they get worn down too, just like objects.
So it is is describing something physical and psychological, through the use of a phrase that evokes an image of physical degradation of things other than a body.
It's a metaphor!