These two questions have a similar meaning, but they are not exactly the same. 'What do you mean?' may sound rude or angry. It means 'I don't understand you' or even 'What you are saying makes no sense/ is unreasonable/ is stupid' 'How do you mean?' is much politer. It is just asking the person to clarify what they have said.