It sounds very unnatural and confusing. The problem is that you've put the negative at the end of the sentence, when it should be near the front (within reason).
"You are not talking to me clearly" is the best way of correcting the sentence. The other option is to replace "not clearly" with another word or expression.
"You are talking to me vaguely."
"You are talking to me in riddles."