Your friend is complimenting you in your ability to communicate in Japanese despite the difficulty. He is saying you're fluent as well. Whether or not he actually believes you're fluent or not, I'm not sure, but this seems to be a cultural habit in Japan. When people praise you, they will do so in such a manner that, to westerners, may seem excessive. It's simply them being polite, however. The common pattern is that when someone is praised, they usually will say "oh, no I'm not not, I only know a bit of Japanese," for example, to express humility. This is the socially acceptable thing to do. In some contexts, to agree with their compliment or accept the compliment in any way may seem rude or arrogant, but that depends on how you say it. You can simply say "thank you, but no, actually I......" So, this way you are showing gratitude for the compliment, but also maintaining modesty.