sounds peculiar to me...Grammatically it's correct. I'm sure of that. but it's not common I think. Using "going to" shows your intention of inviting him/her over and basically, you invite sb to sth. But maybe it's better to say: "let's have a coffee together sometimes" or other ways of expressing it...