That's correct. You can say "let's call it a day" to mean "ok, we've done enough work for one day, let's each go home and rest". You can think of it as "let's call it a full day's work, and quit". "Let's call it a night" is a common variation; to me it means "let's call it a full night's work" (although "work" needn't be taken literally, it could be a full night of fun). It's not literally saying that you've worked all night. More like all evening. "Let's call it quits" also sometimes means that you're ready to quit work and go home, but it can also mean "let's stop arguing" or "let's break up (romantically)" or "let's consider our debts are now balanced out."

Generally you would say that when you are tired and done with something. So yes.... you thought right.