I also had a lot of trouble with these sounds when I was starting to learn American English.

There is definitely some variation in these sounds from person to person and even from one utterance to another, sometimes to the point where they sound identical and only by the context can you tell which one is which.

Generally, I think Phil gave the best explanation as O is very open and U may be closed or muffled, but not always is.

Check these out:

http://www.forvo.com/word/cashew_nut/#en    by rdbedsole

http://www.forvo.com/word/betel_nut/#en       by sdoerr

And here:

http://www.forvo.com/word/not/#en

rkaup (Male from United Kingdom) 's pronunciation can be confused with nut, right?

Well, don't make that "mistake"

You can record how you pronounce the sounds here: http://vocaroo.com/

If we hear how you say them, we can help correct your pronunciation.

The vowel in "cop" is as open as possible (open your mouth and lower your tongue out of the way), whereas the vowel in "cup" is somewhat closed.

In the American accent, the short u sound and short o sound are very different. The short u sound in "nut" and "cup" is more similar to the chinese "阿" while the short o sound in "not" and "cop" is more similar to the chinese "哦".  Not EXACTLY similar but very very close. Hope this helps! You can try those dictionary websites where they provide audio of american accented pronunciations of these words and compare them with each other. Good luck! :)

I can help you but we would need to speak online.I cannot help you to hear it without sound! It is easy to practice those sounds once you can hear the difference.