Most people rarely use fillers on purpose. Mostly they just happen and we are not aware of them until we hear a recording of ourselves. Then we try to cut them out.
As Beth and Victoria say, the occasional "hmmm" or "mmm" in conversation will go unnoticed, or may even be encouraging to the other person to show you are listening. If you are making a presentation or video though, or any type of solo speech, they are not good.