What do you mean? asks for a complete explanation. The speaker did not understand any of what was said, or understood very little of it.
How do you mean? asks in what way what was said is true. The speaker understood the main idea of what was said, but wants clarification of the sense in which it was intended or how it applies to specific situations.