'I've just been to the gym' means that you went to the gym and then you came back. This indicates movement from one place to another, so we use the preposition 'to'. This sentence is not about the time that you spent at the gym. It's about the fact that you have just returned from a trip away from your house your workplace, for example.