You can say 'come and see by yourself', but it means, come and see, but come alone, with no-one else. Come and see for yourself, means come and see so that you will believe me, by seeing with your own eyes, instead of relying on me to tell you.

They're both technically correct, but "Come and see for yourself" is the only 'natural' answer. They'd also both mean different things. "Come and see for yourself" means "Come and see with your own eyes" ie. someone might not believe what other people said they saw, so you might say "It's true! Come and see for yourself!". "By yourself" means "alone". So "Come and see by yourself" really means "Come and see, and come alone". It's a strange way to word it though - if I heard it, I'd probably just assume they meant "for yourself" instead, unless they paused at the right time ("Come and see... by yourself"). It'd be much more natural to put "by yourself" in a separate sentence or clause ("Come and see, and come alone").

You should say "Come and see for yourself".