I think the more common idiom goes, time flies when you're having fun.

You can imagine something passing by you, like, a car approaches from a distance, passes you and disappears. Time, or more specifically a certain point in time, does the same. The time point of one week ago was originally far in future, then came to you and now has gone by. "past" represents this concept of "passing by" as a preposition or an adverb. So you can say "The car drove past me" (prep), "I was going to the post office but continued to walk past it because I was too absorbed in thoughts" (prep), or "Time whizzes past" (adv). Similarly "past" as a noun means what has already gone by, the opposite of "future", what has not yet come.

It means that you do not think about time when you're enjoying yourself. You dont consider for time, you just live and fill happy. I think the word past here means as it means in the next sequence of words - past, present, future.

It´s used to indicate the lapse of time, for example: "the night went past without me falling asleep".