It means "the key element/message/argument/idea". If someone says, "That's not my point", he means, "You have misunderstood me. That's not my key message. That's not the most significant element. That's not what I am trying to tell you." If someone days, "My point is...", he means, "The most significant element is…"

it means that it was not what they meant, it was not what they were trying to emphasise. point is something you are trying to get at, explain it is the big picture, the thing your finger is pointing at. allude, imply, refer, suggest. <synonyms for Point