Gdanning gave a great answer. "I never really noticed" doesn't mean that you never noticed at all, but it means you never paid very much attention (you only "sort of" noticed).
Another way to think about "really" after a negation ("not really," "can't really," "wouldn't really," etc.) is to say that the word "really" makes the negative statement softer. "I don't want to go to your party" sounds rude; you would probably say this to someone you dislike. "I don't really want to go to your party" is much more gentle. It still means that you don't want to go to the party, but the word "really" makes the negative statement less strong.
English speakers use this construction a LOT in order to avoid making strong negative claims. Statements such as "I don't like [X]" can sound very harsh, especially if [X] is something the other person likes ("I don't like the movie you showed me," "I don't like the restaurant you've suggested," etc.). It's more polite to say that you don't "really" like something. The meaning is essentially, "I don't have a strong liking for it" or "It's not something I strongly prefer."