It's not impolite, just very direct. How direct to be depends on the nature of your conversation and your relationship with the person. It also depends on the personality of the person. Some people are sensitive to criticism. Others love sparring and matching wits. Such people can actually relish being challenged.

I’m not sure you’re right about this.