For me the biggest problem is the vocabulary I had to learn (mainly for kanji tests). On the one hand I never heard or read those words in context, we just got a (huge) list we had to know for the tests. There were so many words that I never had time to look for the context in which they are used and a lot of them only started to come up after I started to learn academic Japanese. The other problem is that our textbook always skipped the words that you need for everyday conversations the most (like おすすめ). I literally know double the amount of Japanese vocabulary compared to English vocabulary, but it’s much harder for me to express myself in Japanese. Also when talking in Japanese it happens often that my conversation partner uses a word I have never heard before (that barely ever happens to me with English).
Also some Japanese teachers seem to think that giving your students a set of phrases (which they are of course supposed to learn by heart) or having them read a dialogue out loud replaces any other conversation exercise. Nevertheless I've never met a person which became fluent in any language which this method.
Wow, that must have sound very negative. Sorry about that. Something which I also struggle with (and that my Japanese teachers are really good at teaching) is the pitch accent in Japanese. For examples when I say 橋 and 箸 or ぶどう and 武道 it's sometime very hard to hear the difference.
I absolutely love Kanji and don't have any problem with them even though I can understand that some people might. The same goes for same Japanese ways of saying stuff, which often translates to the opposite meaning like 「日本人はいないか?」= "Are there any Japanese?"