It depends on the context it is used in. Friendly could be to make you think. Angry could be used in a sarcastic way. It can be used to imply you are trying to make someone mad/angry. Often it would be used by a person - You are screwing with my mind! But the saying - You are driving me nuts(crazy)! - would be used far more often.

It could mean "I'm saying things that are making you overthink and be restless"