Maybe it will be easier to understand if you look at what each word means by itself.
"Selbst" is "self", obviously.
Selbstbewusst - "bewusst" means "aware". You are aware of yourself, of your good qualities.
Selbstsicher - "sicher" means "sure" or "certain", you're sure of yourself, of your qualities.
Selbstvertrauend - "vertrauen" means "trust", you trust yourself and your strength.
So you can see, all of these are a tad similar.
As far as using those words appropriately is concerned, I'd say you use "selbstbewusst" as a general term if you refer to self confidence. "Ich bin selbstbewusst" - "I am confident".
You do not usually use "selbstvertrauend" if you refer to your confidence, you actually hardly ever use that word as an adjective. The noun "Selbstvertrauen" is much more common.
"Selbstsicher", however, is interchangeable with "selbstbewusst".