Good question. In your sentence: "Hey, you are in the right path to your answer, but not over by." I don't think you should use "over by." "Over by" is usually followed by a noun. For example: "The bag is over by the window." or "Can you put this over by the table, please?" So making a sentence putting "over by" at the end of it, and not following it with a noun, would sound odd and wouldn't make sense. I also don't think you can add a noun to this sentence after "over by" because even if you wrote "but not over by the answer" it somehow doesn't sound right. I would, instead, say this: "Hey, you are in the right path to your answer, but not close enough, yet."

To communicate that idea, you might say, "Yes, you are ON the right path to your answer, but you are not there yet."

Over by is in a literal sense, you can't use it for abstract ideas, for example: "Where are you right now?" --"Over (here) by the street light, can you see me?" For your example, you would say: "you are on the right path to your answer, but no quite there" Because "right path to your answer" is not literal, it's figurative. (Hope this helps!)

You wouldn't use it in that way. It would only be used to refer to something physical. (As you said, it needs to be some distance away, and next to something). It also would never be used at the end of a sentence. It would always be followed by something else. "Where is the ketchup?" "It's over by the microwave" "The post office is over by the train station" "I left my keys over by the television" Hope that helps a bit! :) (Also a minor correction, it should be "on the right path" rather than "in the right path")