No, it doesn't sound natural. :D It would mean that you physically like rubbing shoulders with someone, since you're talking about one specific person. I think the phrase is only used when referring to a group of people that stand out for some reason be it fame, fortune, or intellect. I don't really see it as negative phrase, but in saying that, it's just not common enough for me to have a clear sense of how it's used the majority of the time.
Here are a couple of examples of it used positively:
- After the conference, a lucky few were invited backstage, where they got to rub shoulders with some of the top minds in the tech industry.
As you can see, I have only been able to come up with one example, so you're probably right :D I think most of the time it does have a negative connotation. Although, strictly speaking, it can be used in a positive sense.