It is okay. But my opinion is that we would probably talk in terms of travel time and not distance, so something like: We usually feel it takes longer to get to a destination than it takes to return from there.

Aya-san, I would phrase it as, "It feels like it takes longer to get there than get back".