Lauwrence is right, 'wil je gaan?' is possible : )
FYI, 'je' is the casual form of jij/jou/jouw.
'Wil je met mij gaan' is not necessarily more natural.
Other options would be:
Wil je erheen (gaan)?
Wil je ernaartoe (gaan)?
I wrote 'gaan' between brackets because it is almost always omitted!
If you are going anyway and you are merely asking the other person if he/she would like to join, it is most natural to ask:
Wil je (met me) mee (gaan)? (I think maybe this is what Laurence meant)
Hope it helps :)