in the US, our well-known skyscrapers (buildings, towers etc) are labeled with "the". Parks, stadiums, etc typically aren't.
English translations of foreign names might follow a different convention, but to my ear 'Let's go to Tokyo Tower' sounds wrong, or it sounds if you are treating 'Tokyo Tower" as a place name or destination - not as a building.