No. You can say that the fruit has a 'leathery' texture, but 'leather' itself only refers to the material produced by tanning animal hides, which typically come from cows.

As far as I know, we just refer to the skin of a fruit as "the skin".