This all depends on how you use it. I think if you use it to point out an objective fact "It would be fair to say that the stars don't really revolve around the Pole Star", it sounds fine (I don't think you would be accused of being a weirdo or dork, but don't take other people's criticisms too harshly, everybody speaks a little bit differently). If you use it to criticize someone else "It would be fair to say that his work was very sloppy." then it might sound a bit smug, because it's an objectifying euphemism. That is, you're giving your opinion, but in language that tries to make it sound like objective fact. If you don't want to sound smug, then give your opinion more clearly: "I'm sorry to say that I find his work quite sloppy." I believe this is more a cultural difference than a linguistic phenomenon. Having lived in China for three years (yes, I know China and Japan are very distinct, but I believe on this point they may be similar), it can be considered impolite to express your opinion too directly, so objectifying euphemisms may be more common. In the English speaking world, the opposite can be true. Too many objectifying euphemisms can be construed as being superior or "high and mighty". Those are my thoughts, but I'd love to hear other people's ideas on this situation.

It's not used often as a statement as it's not logically needed. If you think "James is stupid", you don't need to say "It would be fair to say that James it stupid" - you can just say "James is stupid" or "I think James is stupid". Used as a statment, it's used to show that you have been thinking very carefully about what you are saying (and often used as a joke). I hear it at work in meetings where people don't necessarily agree and we are trying to find agreement. "Would it be fair to say there have been some problems in this project we didn't expect?".