Hello antman,
Your question is correct.
"not really" expresses polite doubt. It is not a definite 'no' ,It is neither "yes" nor "no" as you mentioned.
Do you like this movie?
Not really.
The person doesn't mind the movie,but doesn't like it that much.