it just means doing physical actions that might hurt your body

It is a typical phrase for a job advertisement. The people are saying that the job might require you to pick things up or to bend over and stay bent over for a period of time. Some people have injuries and are in too much pain to do those things. This job is not for them.

I think it refers to the ability to perform basic, simple physical movements. Somebody who can neither lift nor stoop is suffering from physical impairment.