"Have been to" and "Have gone to" are basically the same. It focuses on moving to a place, but also hints that you were doing something there.
"I have been to the Library," works the same as "I have gone to the Library."
"Have been in" is a little different. The focus is on participation or being inside of something.
Technically, it would be correct to say "I have been in the Library" but the meaning is different than "I have been to the Library."
You could use them both to mean "I went to the library and did normal library things (read books, borrowed a book)." But you could also use it to mean simply, "I was physically inside of the Library (I know what it looks like inside there)."
Furthermore, you could say "I have been in America for 5 years," suggesting that you are still in America.
While you would say "I have been to America. I was there for 5 years," if you are currently not in America.