I'd agree. It sounds much more natural with 'two'. It even sounds a bit weird to me without it, as if it's meant literally somehow. Google ngrams supports this too. With 'two' is far more common.

Hi! Native English speaker here! I have only ever heard it said WITH "two." Of course, it's not grammatically incorrect to say it without, but I think the "two" is included for emphasis and to strengthen the idiom's meaning. To sound most natural, I would always include the "two," but maybe someone from a different English-speaking region might have a different opinion.