I didn’t expect that is the past simple (used for a short actions in the past). I wasn’t expecting that is the past continuous (used for a longer actions in the past or habitual things in the past). Hope this helps;)