It's not a very logical response, but it makes sense if I imagine that person B was hoping they were a lot closer to the end point. Person A's statement is optimistic -- we've already come so far, and we're doing very well. Person B's statement is more pessimistic -- what do you mean, we're only halfway there? We've been walking forever!
I feel like it's more about how the travelers feel about the distance than about the exact point in their journey, if that makes sense.
I hope that's a little helpful!