I think “starting from zero” or better “starting from nothing” would both be widely understood. It might be more natural to say something like “I don’t want to start all over again.”

For example, start from scratch. When your project/business/plan failed and you have to start again from an empty place without anything. Good to describe your condition in few words. Or when you have stable good place, but someone offers you to start something new and get risk. So you could answer: "I don't want to start from scratch"

Well, for example, start from scratch. When your project/business/plan failed and you have to start again from an empty place without anything. Good to describe your condition in few words. Or when you have something, but someone offers you to start something new and get risk. So you could answer: "I don't want to start from scratch "

Like many other idioms in English language, this one probably originates from sporting terms. In the days before fancy running tracks, when running a race, someone would draw a starting line by making a scratch across the dirt; hence, the runners start from scratch.