This idiom was used in 1542, when the phrase first appeared, "to go to pot" was to be cut up like chunks of meat destined for the stew pot. Such a stew was usually the last stop for the remnants of a once substantial cut of meat or poultry, so "going to pot" made perfect sense as a metaphor for anything, from a national economy to a marriage, that had seen better days. Early uses of the metaphor were usually in the form "go to the pot."
"Go to pot " is similar to other expressions like "go to rack and ruin" or "to go haywire" Other examples: Our picnic went to pot/ rack and ruin once the rain started.
Things at the company went to pot/ went haywire/went to rack and ruin because the management was lazy.