Good question. In your sentence: "Hey, you are in the right path to your answer, but not over by." I don't think you should use "over by." "Over by" is usually followed by a noun. For example: "The bag is over by the window." or "Can you put this over by the table, please?" So making a sentence putting "over by" at the end of it, and not following it with a noun, would sound odd and wouldn't make sense.
I also don't think you can add a noun to this sentence after "over by" because even if you wrote "but not over by the answer" it somehow doesn't sound right. I would, instead, say this: "Hey, you are in the right path to your answer, but not close enough, yet."