I've heard it used in two different ways. It can mean "little bits of time", as you said. Applied to a person it has a different meaning. If you say "he sometimes has odd moments", it means he occasionally acts in odd (not normal) ways - either in terms of his behavior, or his mental clarity.