"Do you mind" is used 1.) to ask permission do so something or 2.) to request someone do something for you. It's not used to invite someone to do something, or to ask if someone wants to do something.
When I hear "Do you mind having dinner with us tomorrow?", I imagine we already have plans to have dinner another day. For example: "Sorry, I forgot that I have to work next Monday night. Do you mind having dinner with us tomorrow instead?"
So the best way to invite someone is to say, "Would you like to have dinner with us?" or "Do you want to have dinner with us?" Don't forget to say "to have dinner" and not just "to dinner."
I hope that was helpful!