I believe it would be best to say:
'I have a difficult job'' (correct)
'My job is hard'.
To have a hard job seems, to me, to sound like your job is very dense, like a rock is dense/hard. It is more colloquial to say "my job's tuff/hard" or the less formal ''My job sucks."
Either way, the meaning of your idea would be understood so I would not worry about it too much.