I would call a person who is bad at sports "un-athletic." "Clumsy" is much broader. Someone who is clumsy is likely to drop things, or step on their partner's feet while dancing. It is the opposite of "graceful," or "deft," or "agile." "It was very cold today, and I forgot to bring my gloves. When a shoelace came untied, I discovered that the cold had made my hands so clumsy that I couldn't tie a bow."

I can't think of one word which expresses all your ideas vividly. You could say that they are "not sporty".