'that' refers to a quantity that has been stated or presented, i.e. someone is pouring wine in a glass for you, and they've poured too much, so you say 'not that much'.
'not so much' would be said before they've started pouring.
however, in practice, the two are used pretty much interchangeably.