I agree, but also it has a negative connotation of being cheap or miserly. You would not use it to praise someone for being careful with their money.

A miser. It means someone who does not spend enough money. 

"He's such a penny-pincher that he won't replace his tires until they are bald. Someday he's going to have an accident."

Somone who is careful with money in a good way is "thrifty" or "frugal."

Someone who is very careful not to spend much money.

That reminds me of the Johnny Cash line in 'To Beat the Devil'

But it'd been of month of paydays since I'd heard that eagle scream.

I've never heard it in real life, but there is an expression "to squeeze a quarter so tight the eagle screams." 

I always thought the lyrics to "Nobody Knows You When You're Down and Out" said "If I ever get my hands on a dollar again/Gonna hold on to it 'til the eagle screams," but the Internet seems to think it is "'til the eagles grin."