I hate the American tipping system. I think it's ridiculous and annoying. I've heard the argument that it's to supplement workers' wages but, in that case, I think workers should be paid more for the work they do and not have to reply on the gratuity of others. ALSO, I've had the same problem you have with taxi drivers being annoyed for not getting "enough" of a tip when, to me, a tip should only be given in gratitude if someone has gone beyond what their job requires. If your job is to get me from A to B and I've paid you the cost of the service, I should not *have* to tip (and especially not if I was treated rudely which has happened to me so often!). Tipping is optional and the only person who determines IF an how much to tip should be the customer. siiiigh. But that's the end of that rant. Hahah
I don't think you should have to tip any more than you want to. I think any systems reliant on tips to pay their workers should be fixed and employers should be made to pay their workers properly. And I think that driver was rude to you and got more than he deserved. 

In Japan, tips are not expected, not given and, even if given, not accepted.  Service is in general near- perfect.  Now how do the Japanese manage this?

Just to add to Antonio's comment above "If a restaurant server is mostly oblivious to my presence I will still tip the minimum 15% because I know that the financial system is setup that way in the US." I wouldn't tip at all, unless they are swamped there is no reason they shouldn't see to your needs, and that 15% is assumed and not fair. You work hard in your job, so why should you pay 15% to someone who isn't doing their job right? To me it comes down to the notion of no one should be paid a wage that they can't live off. Seatle raised the minimum wage to a living wage, and they seem to be doing alright. 

By the way, the next time you are in San Francisco you can get a ride from Uber, the car service. Uber was started in San Francisco and tipping is not expected. The payment for service is made electronically without an option for adding a tip. However, some people do tip Uber drivers in cash.

When service is extremely rude, some people decide not tip, even in a restaurant and in disregard of the convention. I have heard of people leaving a scribbled complaint on the bill instead of a tip. "No tip for you, soup nazi!"

On the other hand, US residents tend to make minor adjustments in the tip, depending on the quality of service. If a restaurant server is mostly oblivious to my presence I will still tip the minimum 15% because I know that the financial system is setup that way in the US, and servers get a low hourly wage. However, I will tip more than 20% if the service is particularly good. You will find that one benefit of this tipping culture is that restaurant servers smile a lot and are very friendly to customers. I've heard many comments about that from foreigners visiting the US.

In some states in the US, the minimum hourly wage has been raised and some restaurant owners are raising their prices to compensate and discouraging high tips. This is a new and untested phenomenon.

You were OK giving him $3. He may have wanted more, but $3 is not an unreasonable tip for one bag. Many people would give only $2. I can tell you for certain that some passengers would carry the bag themselves and give nothing, ignoring the grumbling.

It's true that some service workers will try to extract as much as they can from their customers, but most are polite and accept what you offer. I live in San Francisco, so I know that the drivers of shuttle buses at SFO are particularly aggressive. Sometimes the shuttle bus coordinator will even give a little speech to passengers when the shuttle is booked about how the drivers need tips. Frankly, this feels like extortion to me and I find it offensive. I tend to tip less when I'm confronted with this practice.

I don't like tipping either, I would rather pay a flat rate, but in the US it is expected for some things such as restaurants (definitely, 15%-20%, with 15% becoming less common) and services involving hotels and transportation.