Consider: Are you hungry? Yes. (I’m hungry) No. (I’m not hungry) Aren’t you hungry? Yes. (I’m hungry) No. (I’m not hungry.) The answers are unambiguous. The statements in parentheses aren’t necessary, That is: ‘Aren’t you hungry?’ = Are you hungry? +(I think you are, or I think you should be) On the other hand: ‘Are you not hungry?’ (Without using the natural contraction) is an abnormal phrasing in speech. It’s correct. But because it’s rare, a simple ‘Yes’ or ‘No’ answer becomes ambiguous here. The meaning in speech also depend on the spoken emphasis. For example, ‘Are you NOT hungry?’ Might be interpreted as ‘Are you full?’ So unambiguous answers are I’m hungry. I’m not hungry.

it's not "No", it's still expresses agreement and you can say both "yes, you are right, I don't have to wait" and "no, no need to wait" they mean same absence of need

In colloquial speech, yes.

In casual speech, yeah means yes and nah means no. English, unlike some other languages, uses polarity-based answers. Example: (1) ‘Is he not coming?’ a. Yes. (‘He is not coming’) b. No. (‘He is not coming’) Answer a is correct for languages that use truth-based (agreement-based) answers. Answer b is correct for English and other polarity-based languages. So: "You don't have to wait here now? No/Nah. I don't have to," is correct. "Yes/Yeah. I don't have to" is a mistake because it violates the polarity-based answer system. Almost all of my students come from truth-based/agreement-based languages and have to be taught that English is polarity-based. Global English has many variants, so some native speakers may use truth-based answers.

The rule is, if you agree to something (like with yes) you can say yeah. Its like nodding the head. So if you have someone asking you a question like: A: You don’t have to wait here anymore? B: Yeah/Yes/ Exactly , I don’t have to!