It means "We're all tired and we've done it enough."
Imagine that one day's work is supposed to last until some time, perhaps 5 p.m. Imagine that it is hot, or that the work has been especially tiring, and that the leader thinks the group could go on working longer but wouldn't really achieve anything useful. "Let's call it a day" means we haven't done the normal number of hours, but let's call it a full day's work anyway.
It could be either a positive statement--we've finished what we needed to get done--or a negative one--we aren't achieving anything.
"We've been negotiating for hours, but we just aren't making any progress and we're just getting tired and angry. Let's knock off early and call it a day, and try again tomorrow when we are fresh."
"Really, you got it all done? That's excellent, I think I'll just call it a day and let you go home early."
A quick Google search on Project Gutenberg texts turns up some examples:
1) "Let's call it a day's work, and knock off," chimed in Waldo.
2) She glanced at her wrist. It had been over four hours since we'd started. "I am rather tired," Jack said. "And hungry, too. Let's call it a day and go get something to eat."
3) There was no chance to loaf, for when a marine looked over his shoulder he could see the French picks going for dear life down at the other end of the trench. At four-thirty the men were told to call it a day.