It's difficult to evaluate outside of context, so I can't say whether the "less" is appropriate or not.
But I feel like you could supress either "about" or "whether" and you have to add the verb refering to "clothes" as well.
Also, depending on what you mean, you should specify what kind of clothes you are talking about, or "popular" may not be the best choice, since it will express that those people actually don't mind whether wearing clothes is popular or not: they might just not wear any.
It could go like this:
"they don't care about this type of clothes being popular or not."
"they don't care whether these clothes are in fashion or not