I believe it is correct. "Most of the time" in this case is the main emphasis of the sentence.

It's perfect. The lovely thing about the simplicity of the English language is that you can say things any number of ways, and your way is just fine. The comma after "time" could be dropped and the sentence would lose no meaning.