My impression is that "go to" is a specific place, usually. "Let's go to the store." "Let's go off to the store.", does not sound right. "Go off" is usually an undetermined place. "Let's go off into the forest." "Let's go into the forest.", sounds fine too though.

Overall, the two statements mean relatively the same thing. However, “to go to somewhere” indicates a focus on the final destination. “To go off somewhere” indicates a focus on leaving the original location (understood: to end up at the other location). “To go off” can also indicate a temporary absence (which is understood in context). For instance, “Cathy went to the office” focuses on Cathy’s office, assuming she will stay there, while “Cathy went off to the mailroom” May be understood that Cathy will be coming back from the mailroom. I hope this helps, and I hope other native speakers/teachers can chime in! I gave it my best :)

“To go off somewhere” assumes that you’re leaving wherever you are now.