Dan Smith is true that too many fillers is not only a suffering for a listener but also a deduction in the mark of a test. Supposedly a good speaker had better eliminate the usage of fillers as much as possible. For a non-fluent speaker, that might be a huge challenge not to use fillers while they are speaking, but it can be reduced to the least degree by constant practice.

I consider even some native speakers cannot exempt using fillers at all, but indeed, too much redundancy is very bothersome for a listener to concentrate on. 

Toastmasters is a self-help club for people who are trying to improve their public speaking ability. (It is mostly a U.S. organization, although there are chapters worldwide). Toastmasters teaches speakers not to use filler words and noises. When you give a talk, three judges take notes: the timer, the grammarian, and the "ah-counter." The timer checks whether your talk runs too long or not long enough. The grammarian watches for grammar and English usage problems. And the "ah-counter" literally does that: counts all the ah's, um's, er's, well's, like's, really's, other meaningless sounds or words that are just used to fill gaps. Toastmasters teaches you try to eliminate them completely. I think Toastmasters is right.