"Let's go around." means you could be anywhere by a certain place and the "around" is anywhere in a half circle kin of.
"Let's go around the back." means you are in front of a certain thing and you want to go behind. And it basically means with "the back" part is you are going in somewhere.
If you would have said, "Let's go around back." means you are more general about where you are going but you are still going behind.
***You can replace "back" with "front" too, only if the situation is reversed.
Hope this helps^^ Have a nice day!