They are pretty similar:
Bankrupt is usually specifically when you actually file with the state and is used pretty frequently in common language, even as a figure of speech: "With these gas prices, I'm going to go bankrupt." (Not literally)
Insolvent means you can't pay your bills usually and is not used very often.
Bust is a slang term in this situation: He wasted all of his money and went bust.