That would be OK. You generally say this when something subtle somebody does or says bothers you for some reason you find difficult to define. For instance, if a waitress were to say goodbye in a sarcastic tone of voice or throw your change at you, she would likely "rub you the wrong way," even if you she were otherwise very polite. Basically, it refers to something specific that bugs you or gets on your nerves.

"Every time I meet her she just rubs me the wrong way"?= "Every time I meet her she really annoys me" Other examples: He raved on and on at the conference and after a while it rubbed me the wrong way. When my students lie to me about not handing in their homework, it just rubs me up the wrong way.