The easy answer is to say that the food has "spoiled" or that it is "no good," because these can applied to any kind of food.
"Ugh, the milk has spoiled, I need to throw it out."
"Ugh, those peaches have been sitting around too long, they're no good any more."
In the United States, "stale" is usually applied to baked goods and means that they have dried out and lost flavor, but are perfectly good to eat. "Rotten" is applied to fruit and vegetables and means that parts of it have become soft, changed color, grown mold, are starting to ferment, etc. Butter and cheese become "rancid." Meat becomes "rotten" or "putrid."
I guess, to paraphrase Tolstoy, fresh foods are all alike, but every spoiled food is spoiled in its own way! (That's intended to be a joke).
P.S. Just to confuse the issue completely--I didn't learn this myself for a long time, probably from reading one of Dick Francis' horse racing detective stories, but--this usage only applies to horses and camels!--"to stale" means "to urinate!" Few English speakers would know this usage unless they belong to the world of horses.