There's also a grammatical difference. 'Clothes' is a plural word. It has no singular form. Shirts, dresses and jeans are clothes. But try to complete this sentence with one word: "My blue velvet skirt is my favourite ........." Favourite what? Not clothes. Not cloth. Not clothe. Those words don't fit. But you can say 'My blue velvet skirt is my favourite garment'. We don't use the word 'garment' very much, but it is useful, because it has a singular form.

Basically, your right ???? garment seems more old fashioned and is less commonly used.