I think this may be grammatically correct, and I think it may depend on the exact context, but the more common phrase would be "I'd be honored if you'd join me for dinner."