"Or so they say" means "so I hear", "so I'm told", or even simply "apparently". It's not a definite fact you're stating, just an opinion. "Or so they say" is used to make this more evident.
"I'm not much of a romantic, apparently". You don't literally need a second opinion to use "or so they say".