It means to do physical work/manual labour

It can depending on context. Working with one's hands is generally harder than working with a machine, etc, so saying something like "He deserves a nap. He works with his hands." Or some better way to say it. lol Otherwise, it just means that you work with your hands. This could be a construction or any other "hands-on" job.

hard labor

I supose it's same as work with bare hands. Or do something alone. Now i need to choose :)

It literally means to use your hands. LOL. I think it refers to people who do the actual work, as opposed to those who do planning, or managers.