Excuse me, but I do not believe this concerns you.
I beg your pardon, but this is a private matter.
I am really sorry, but we wish to have this conversation alone (or, you could replace "alone" with "in private").
There are many other ways to say this. Unfortunately, to many people, being told something is not their business, no matter how polite you are, will seem rude to them. That is their problem, not yours.