Both sentences are grammatically correct, but they have slightly different meanings.
"The cake isn't what I wanted" means that the cake did not meet your expectations. This could be because it did not taste the way you wanted it to, or it did not look the way you wanted it to.
"The cake didn't turn out how I wanted" means that the cake did not turn out as you had planned. This could be because you made a mistake while baking it, or because the ingredients you used were not what you expected.
In the context of your sentence, "The cake isn't what I wanted" is probably the better choice. This is because you are simply stating that you did not like the cake, without specifying why. If you wanted to be more specific, you could say "The cake was too dry" or "The cake didn't have enough flavor."
However, if you wanted to explain why you did not like the cake, then "The cake didn't turn out how I wanted" would be a better choice. This would allow you to explain what went wrong with the cake, and why it did not meet your expectations.
Ultimately, the best choice of sentence will depend on the specific context in which you are using it.