No, I don't mind. I usually don't even notice it, unless it's excessive. I does bother me if the speaker is giving a speech or seminar. Then it can be distracting.

This filler words are completely natural, and every language has them. There is not a single native English-speaker in the world who does not use them constantly when speaking.

Well, for me it's always better and more enjoyable to listen to someone more confident that has a more extensive vocabulary, but since almost everybody use such expressions, I guess it's better if we don't mind, otherwise we will rarely enjoy a conversation :P

I really don't mind. I use that too.

I don't mind them, but as for "often", it can be very painful!