The literal sense of being "quick" when you are to slow means that you need to hurry. So whether you need to hurry because you are late or hurry in completing your activity, you still need to be quick in both/ you'll still be in a hurry.

We have to hurry.

It means you need to act quickly. How or why you need that is an issue not discussed in this sentence.

"We have to be quick" in this sentence means "We have to hurry because we might be late for something." It implies that time is limited, and they need to act fast to fit everything in. It’s not about being physically slow but rather about moving quickly to stay on schedule.