"No" leaves no room for argument. It is a nice, clear, simple answer. However, depending on the situation, it can also be abrupt and even start arguments.
"Not really" essentially has two meanings. Most accurately, it means "not quite no" or "almost no".
Eg. Do you want to make dinner tonight? Answer: Not really, but I will anyway...
Eg. Have you started your book report yet? Answer: Not really, I just chose my book...
In common usage, though, it's usually a polite way of saying no.
Eg. Does this outfit look good on me? Answer: no. Polite answer: not really...
Eg. Do you want to order McDonald's for dinner tonight? Answer: no. Polite answer: not really...
In general, if someone asks you a question and the answer is "no" and you want to avoid confrontation or a potential argument, it's better to say "not really." If you DO have a strong opinion and are comfortable stating it (or it's a trivial matter - "do you like the colour green?") then "no" is a better choice.