I'd say the difference is the degree of emotion or apathy about the subject.
If something didn't bother me, I would be neutral about it. I might wear the slogan or logo or I might not.
If you say, I guess I don't care about it, that would make your listener believe you have a negative emotional reaction to it. Almost the same as saying, I'm offended or I'd rather not wear slogans or logos.
In this case, I believe the correct response would be the first one. If you have never really thought about it before, it would not bother you.
To say you don't care about it, sounds like you don't like slogans or logos, rather than you don't have an opinion about wearing them.