It tends to be used ironically or sarcastically so you would say it about the unknown small town (which has nothing else famous about it) but not about New York which obviously has a lot of famous aspects. The 'claim to fame' itself might be famous or well known but would be the only significant thing about that person or place. My claim to fame is leaving helpful answers on italki ;-) - Yes David's claim to fame was once hitting a home run in high school - Yes Kim Kardashian's claim to fame is being on TV - Yes (she is legitimately famous but you are belittling her by calling it a 'claim to fame' not a substantial or long lasting fame) Picasso's claim to fame is being a painter - No