We are sensitive to the direction in which people are looking.
"Staring" means to look in a fixed direction, without shifting the eyes, for a long time. Usually, when someone is staring, they are staring at something. Often we can tell what they are staring at. Usually it is a person, or something that is attention-getting. "The baby was staring at the moon." "The woman was staring at her cell phone."
Sometimes, especially when someone is thinking, someone will stare, but you cannot identify anything they are staring at. A common phrase for this is "he was staring off into the distance."
In your case, it sounds as if the person's gaze was not as far up, so he did not seem to be staring into the distance. But he wasn't obviously staring at anything. His direction of gaze was in front of him. But you could not see anything that was in the place where he seemed to be staring. He was just staring in front of him, but not staring at anything.