Both are correct. In this case = the entire situation - in the case that he is sick. In this state = his state or condition - the state = sickness

A conversation could go something like... Person 1: "He's sick" Person 2: "In THAT case, he can't travel" Person 1: "No, not in this state"

Natural is: In his condition, he can’t travel. (‘this’ instead of ‘his’ requires his ill condition be obvious in front of of the speaker) Also possible: A: Jim’s running a fever and vomiting. B: In that case, he can’t travel. I suppose it could be theoretically possible to be well enough to travel but too sick to “go on a journey” but it’s not a very clear way of speaking since ‘going on a journey’ is rare.

Take this from an native English speaker, in this state, is the correct choice option, try not to over think it, he is sick, is the only context needed to know that he is not in the best state to go on a journey. In this case, is not the correct answer as it goes over the rest of the sentence as if there was more to be said. Again don't over think it, you'll get the hang of it

The first sentence is a bit unnatural; something like, “In that case, he can’t travel.” Although you could use the phrase, “on a journey,” this would be more literary/poetic, rather than conversational. Regarding the second sentence, it would be much more natural to say something like, “He can’t travel in this/that state.”