I don't know whether there's a rule for the difference between the two, but the difference in the way I use them is that 'in which case' can link together two clauses in the same sentence, but 'in that case' cannot do this by itself. 'It looks like it might rain today, in which case I'll need to wear my coat.' 'It looks like it might rain today. In that case I'll need to wear my coat.' However, if you add a conjunction before 'in that case', it feels natural to link two clauses in one sentence: 'It looks like it might rain today, and in that case I'd need to wear my coat.'