When you use "I visited", you must tell us when. It's talking about the past.
"I've been to..." means it happened sometime in your life experience. You don't indicate the time of the action when you use present perfect.
It has nothing to do with sounding casual or formal. Both forms are standard, common English.