Hi there, that's actually a very good question!
'A lesser man / woman / person' can be quite a strong word and can mean 2 things:
Someone who is not at moral as yourself.
Someone who is not as good as yourself in some way.
Of course, the phrase can suggest that you think certain people have a higher value than others. A more polite way of saying the phrase with the two meanings I have mentioned could be:
Someone who is not at moral as yourself: 'Someone without such high moral standards...'
Someone who is not as good as yourself: 'Many others..'
If you want to talk about other males specifically you could say: 'Most other guys'
For example: 'I can drink 12 beers in one night, most other guys wouldn't be able to do that.'
I hope this helps, there isn't really any equivalent which carries exactly the same meaning as 'a lesser man' even though the phrase can be very strong. The only way you can say the phrase in a more polite way is to sacrifice some of the original meaning.