"I wasn't talking about you" If you use "speaking", it sounds a bit more formal. "I wasn't talking about you" sounds much more natural/colloquial.
"I didn't mean you" is almost the same depending on the context. For example:
"I hate it when people chew with their mouth open"
"But I was chewing with my mouth closed!"
"I wasn't talking about you." OR "I didn't mean you." are basically the same in meaning in this situation.