I'm going to try to explain the literal meaning of the phrase. It refers to a person who engages in some kind of trickery, or some other kind of clever social or psychological manipulation. One can imagine this conversation:
"Do you think that so-and-so is a con man (swindler, fraud)?"
"It takes one to know one."
A con man succeeds because he is able to fool honest people. Therefore, an honest person will find it hard to detect a con man. However, a person who is a con man himself knows all of the tricks himself, and more likely to be able to detect them in another. You need to be a con man to recognize the tricks of a con man; it takes a con man to detect a con man; it takes one to know one.
As others have mentioned, it is also used as kind of insulting retort. "It takes one to know one" can mean "you are the same kind of bad person you are accusing someone else of being."
A related expression is "it takes a thief to catch a thief."
In a negotiating situation, if someone thinks you are trying to deceive them, they might say, in a mostly-joking way, "Don't try to con a con man." That is to say, "don't try your negotiating tricks on a person who is good at negotiating tricks."