Of all time is used when saying something is the best ever. Michael Jordan is the greatest basketball player of all time.

Of all times is usually used when saying something happened at a wrong time. You could have visited Damascus all these years, but you picked now of all times when there is a war going on!

Correction-   We do say now of all times like Matty says below.

Of all time means ever.

He is the best boxer of all time.

He is the all time best boxer.

Of all the times that...

Of all the times that my aunt has visited, this last visit was the most annoying.

of all time is the correct idiom/expression that we use.

of all times = what does it mean? what? times? how many times are being referred to.?

even my British spellchecker does not like of all times