Yes, you can say this - but only in the very narrow context of clothes shopping, and only when you are talking about garments that are sized according to the general categories named 'small' 'medium' and 'large'. In that situation, it would be clear that 'I'm a small' meant that you were saying which of these sizes was yours.
In any other situation, it would be seen as ungrammatical and meaningless. Context is everything.