Hi Tsu, Use "does it matter to you". I don't find the phrase impolite (please, anyone disagree if you think otherwise!), but a very polite version may be, "would it matter to you...?" This makes the question sound theoretical, and less immediate than "does...?"

Hi Tsu, I would use the phrase 'Do you mind?' instead. Example; Do you mind if I leave early? Do you mind if I check my phone messages now? Do you mind if I smoke here?

"Does it matter to you?" is correct. Like: "Does it matter to you that your son is being expelled from school?" "Would it matter to you," regret to say, is not exactly the same, as "would" denotes a future event here, tied to a conditional; like: "Would it matter to you if your son got expelled from school?"

Hell Tsu, No ,it sounds alright, however it depends in what context it is said and how it is said; the tone plays a role in determining the intention of the speaker.