Ofcourse it is difficult to remember new words, but the most challenging thing for me is the grammar.  Since Japanese grammar is totally different from English one, I am having the hard time to speak/write in the right grammar.