You don't speak a speech. You either give or make it, so you could say either 'The day the speech was given' or 'The day the speech was made'.

The usual collocation is "to give a speech", so it would be better as "the day the speech was given".