This is the basic difference:
'Hurt' means to cause PAIN - either physical or psychological.
'Damage' means to harm or spoil something. This causes some kind of negative CHANGE that has to be repaired, or is permanent.
Objects are damaged. If you damage something you break, tear, dent, split, chip, it or make it unusable. You can damage clothes, shoes, an appliance, a car, a piece of furniture, and so on. Objects cannot feel pain, so you can never hurt an inanimate object! As Colin says, plants come under the category of inanimate objects : you can damage a plant - if you tear its leaves, for example - but you can't hurt it.
Living things can be hurt. If someone cuts your skin, this hurts you physically. If someone says something unkind, this hurts you emotionally.
It is possible to damage a person, but this is a serious long-term effect. We could say that a person is psychologically damaged if they have suffered a serious trauma that they can't recover from, or which needs therapy. Likewise, you could say that a part of the body is damaged if it has an injury that does heal on its own .e.g. 'I have a damaged knee ligament from a a bike injury when I was a child.' This suggests that the condition is permanent.