I'm pretty sure it means that you would be hitting him with the back of your hand (the part that's not the palm of your hand). You could also say, "he deserves to be hit with the back of my hand." Some people call that kind of hit a "smack," probably because of the sound it makes.

"He deserves to be hit".

This describes a strike using the back of your hand. It's also a belittling hit (ie. the person who is hit is insignificant) because it looks like a "brushing-off" movement. If you "show someone the back of your hand", then that means you hit them in this way. If you seem to praise someone but you're really criticising them, then you've given "a backhanded compliment".

It means that fool deserves to be slapped.

It is a figurative language, you never take the meaning literally in such cases, in this context the phrase” he deserves the back of my hand” either means that he deserves to be hit (as in slapped), or simply he doesn't deserve anything at all.