If you actually do mind, it wouldn’t be sufficient to say “Yes”.
Yes, I do mind.
Yes, in fact I do mind.
Note also that “do you mind?” has a different use as well.
If I’m talking and someone interrupts me, for example, I could say “Do you mind?” (I was talking and I’m annoyed that you interrupted me. I want you to stop talking so that I can finish.)