In my interpretation, the speaker is indicating that a person so sweet, analogous to sweet morsels harmful to teeth, is detrimental to dental health.

It implies that you are are so sweet, that you would give them a cavity if they ate you :,) It says the word "sweet" because if you eat too many things that are full of sugar and sweet tasting, then you will get a cavity in one of your teeth!