집에 가자. literally means "Let's go home." while "돌아가자" means going back to the place where you come from.