"Of" is not a good preposition to use here. It makes sense to speak of the "purpose of coming here" but that purpose does not belong to you. Instead, the purpose belongs to "coming here". It is "coming-here's" purpose. Your sentence is confusing because it makes the "purpose" the property of two different owners: "coming here" and you. You can say it better this way:
"Seeing him was the purpose of coming here. That's why I came."
If you had used "for" instead of "of", you would have avoided such confusion:
"To see him is my purpose for coming here".
Now, "purpose" becomes a synonym for "reason". The purpose clearly belongs to you. It is yours. "For coming here" now simply describes what sort of purpose it is.