Your friend is complimenting you in your ability to communicate in Japanese despite the difficulty. He is saying you're fluent as well. Whether or not he actually believes you're fluent or not, I'm not sure, but this seems to be a cultural habit in Japan. When people praise you, they will do so in such a manner that, to westerners, may seem excessive. It's simply them being polite, however. The common pattern is that when someone is praised, they usually will say "oh, no I'm not not, I only know a bit of Japanese," for example, to express humility. This is the socially acceptable thing to do. In some contexts, to agree with their compliment or accept the compliment in any way may seem rude or arrogant, but that depends on how you say it. You can simply say "thank you, but no, actually I......" So, this way you are showing gratitude for the compliment, but also maintaining modesty.

I totally agree with Martyn. This is only my opinion, I often say "すごいねー" with not mean much in accordance with the circumstances. It is kind of listening sound in some cases. I am sure that he was just trying to be polite.

I am learning Japanese myself and was interested in this. Having spoken to a friend about it, "だもん” is used like "です" but I imagine less formal. I would have interpreted his comments as something like "Yes, Japanese is tough, but you are brilliant at it" and I think he is basically encouraging you.