gym is the abbreviation of "gymnastique", which is the name of the sport.
So "aller à la gym" means that you are going to practice the sport called "gym" (gymnastique), not that you go to the room in which you do the sport. To say this you have to say "aller au gymnase". You can't use gym because it doesn't mean "gymnase".