Its "i am not that close to you" and yes you can use it!

I think of 2 different things when I hear "we're not that close" and "I'm not that close to you" "We're not that close" means we are not very good friends. We do not have a strong relationship. I have never heard someone use the phrase "I'm not that close to you" ...but if I heard it, I would think of physical distance, and not a relationship/friendship. If you are talking about a friendship or relationship, I think it is better to use, "We're not that close."