This all depends on how you use it.
I think if you use it to point out an objective fact "It would be fair to say that the stars don't really revolve around the Pole Star", it sounds fine (I don't think you would be accused of being a weirdo or dork, but don't take other people's criticisms too harshly, everybody speaks a little bit differently).
If you use it to criticize someone else "It would be fair to say that his work was very sloppy." then it might sound a bit smug, because it's an objectifying euphemism. That is, you're giving your opinion, but in language that tries to make it sound like objective fact. If you don't want to sound smug, then give your opinion more clearly: "I'm sorry to say that I find his work quite sloppy."
I believe this is more a cultural difference than a linguistic phenomenon. Having lived in China for three years (yes, I know China and Japan are very distinct, but I believe on this point they may be similar), it can be considered impolite to express your opinion too directly, so objectifying euphemisms may be more common. In the English speaking world, the opposite can be true. Too many objectifying euphemisms can be construed as being superior or "high and mighty".
Those are my thoughts, but I'd love to hear other people's ideas on this situation.