It is important to remember that the same action can be described in different ways (tenses & aspects) so it is NOT the action that determines the tense but the image that the speaker wants to communicate about the action.
Here we have the speaker describing finished TIME, so a PAST TENSE makes sense.
Now which ASPECT between CONTINUOUS and SIMPLE?
The Continuous or Progressive describes an action like a VIDEO, something we imagine 'happening', (or temporary or changing)
The Simple aspect describes it 'simply'. As an event, a fact, permanent - something that cannot be interrupted - like a PHOTO.
So here do you want the listener to imagine a singular fact "He shouted at me" or a continuing action "He was shouting at me" - I think with the "ALL DAY" the speaker wants to emphasize an ongoing action, so the Past Continuous would communicate the image the speaker would like us to imagine.
Note how 'all day' could be used with the Past Simple, for example:
Why did he stay here all day?
Here the speaker is interested in why the event (Simple) happened, not why the action (Continuous) was happening.