In colloquial speech, sometimes it is left out when the pronoun is "you" (usually, the helping verb is also left out):
"(Do you ) Want to go to the movies?"
"(Are you) Going somewhere?"
Also, sometimes it's left out in a response to a question when the context is obvious.
"What's he doing?"
"(He is) Going to the movies."
However in writing and less informal speech, the pronoun should always be there unless it is, as you said, imperative.
Hope that helps! :)