Both are correct and express the same idea. There is no reason to dissect the grammar too diligently.

They both express almost the same idea, though the first expresses that you enjoy speaking with them in general. While 'I'm happy that I have talked with you' is more of an ending phrase in a conversation.