both of these are essentially the same in colloquial setting. Examples: "He was lost on the road and he was sure he had passed by that tree at least three or four times / He was lost on the road and he was sure ha had gone past that tree at least three or four times" So there's little functional difference between the two really. The difference between them might be seen as more idiomatic. we would prefer passed by over gone by in a few situations, or vice versa.
I feel like time has passed me by while i sit here at this job that i hate (here i'm separating the phrase with the object pronoun "me") / He felt like too much time had gone past in that job he hates.
You'll see here it's easier for me to use "passed by" in the present perfect simple but i can't use "had gone past" as easily in the same way, i had to keep in in the past perfect simple instead to make it sound okay.
I hope this helps. Let me know if you have any more questions.