see where he's going
In an American movie I'm watching, a homeless laborer, George, wasn't allowed to stay overnight on the construction site he works. Then another laborer, Frank, approached George and said: "You need a place to stay? Justiceville is over on Fourth Street. They got hot food and showers. I'm going that way if you want me to show you." George didn't say anything (i.e. he refused his offer), but followed him, anyway. After some time, Frank stops at an intersection.
Frank: I don't like nobody following me, unless I know why.
George: Well, I don't join up with anybody, until I see where he's going.
After these words, Frank smiles and they go together to the Justiceville.
Could you please explain to me "until I see where he's going", please? I read it literally: "until I see (with my own eyes) the place where he's heading to", but I suspect I'm wrong and it is used here in some figurative sense, which I cannot grasp.