'All the time' flows better than 'all of the time' when you speak, so that is the one we all tend to use. Some people may pronounce the 'of' when they are annoyed or talking to small children, but in general don't worry about it.

I agree with Thuy Bien. 'All the time' is by far the more common expression. One rarely hears 'All of the time'. I wouldn't worry about the version with 'the', if I were you.

Honestly, I don't think there is much of a difference at all. It's sort of like how some my say "get off it" and some might say "get off of it". The latter is more American. It might just be a locality thing or whatever sort of feels the most natural at the time.

As I know, we only use : All the time