Yes, you can say this - but only in the very narrow context of clothes shopping, and only when you are talking about garments that are sized according to the general categories named 'small' 'medium' and 'large'. In that situation, it would be clear that 'I'm a small' meant that you were saying which of these sizes was yours. In any other situation, it would be seen as ungrammatical and meaningless. Context is everything.

You can say this when you are shopping for clothes or shoes. It's very casual oral English, not to be used in writing.

Correct! When someone asks us our shirt size, for example, it's usual to say "I'm a small/medium/large etc.". Otherwise we must follow the adjective with a noun to give it context i.e. I'm a small PERSON/GIRL/BASKETBALL PLAYER etc.