괜히 is not modifying 아니다 directly in your example. It works on 작은.
So it can be broken down as: [키가 [괜히 작은] 것]이 아니다 = It is not [that/like they are [short for no reason]].
괜히, or 공연히 (same meaning but more formal) means "for no reason" or "unnecessarily". It describes something that happens seemingly for no reason or when you regret having done an unnecessary thing that made matters worse.
- 왜 괜히 제게 화를 내세요? = Why are you getting angry at me (when I didn't do anything)?
- 안 되는 걸 하느라고 괜히/공연히 시간만 낭비했다 = We just wasted our time trying something that can't work.