I agree, but also it has a negative connotation of being cheap or miserly. You would not use it to praise someone for being careful with their money.

A miser. It means someone who does not spend <em>enough</em> money. 

"He's such a penny-pincher that he won't replace his tires until they are bald. Someday he's going to have an accident."

Somone who is careful with money in a good way is "thrifty" or "frugal."

Someone who is very careful not to spend much money.

That reminds me of the Johnny Cash line in 'To Beat the Devil'

<em>But it'd been of month of paydays since I'd heard that eagle scream.</em>

I've never heard it in real life, but there is an expression "to squeeze a quarter so tight the eagle screams." 

I always thought the lyrics to "Nobody Knows You When You're Down and Out" said "If I ever get my hands on a dollar again/Gonna hold on to it 'til the eagle screams," but the Internet seems to think it is "'til the eagles <em>grin</em>."