If you want to ask this, it is much more polite to say 'Would you like to have dinner with me?'

Do you wanna have dinner with me? Would you like to have breakfast with me? Do you want to have lunch with me? No need to add an “a”. I think this is because you are talking about “dinner” as a concept. But the other answer that said it is sort of connected to the verb sounds right also. Here is an exception: If you are asking a friend to meet for a drink, you need the “a”: “Do you wanna get a beer?”

Think of "have dinner" like a two word verb. I'm pretty sure korean has similar verbs, where another word combines with "hada" to form a verb. For example: sarang hada. In this case "have" is comparable to hada. I'm pretty sure this isn't the technical way to explain it, but it might help.