This expression goes very well with the definition that you found in the dictionary. When you say, "What nerve!" you're expressing displeasure/surprise because of the amount of audacity that the person has shown. The person is shameless. A more natural expression, at least in American English, but with the same meaning, would be "He's got a lot of nerve!"

I've not heard of this expression, rather "you're getting on my nerves". as you've eluded to correctly, it denotes a sense of annoyance. hope that helps