The answer might sounds strange, but it all depends on where are you from and what is your mother tongue. For most people on this planet, Polish and Lithuanian languages are the hardest to learn because of earlier mentioned 7 cases, plenty of exceptions in both written and spoken aspects of Polish language and a very difficult pronunciation. However, for people from Russia or Slovakia learning polish should be relatively easy due to the fact that our languages are somehow similar.
Not once and not twice I tried learning Japanese and I failed. Why? Because of the difference in constructing sentences. Polish, English, Spanish and even Chinese are so-called SOV languages (Subject + Object + Verb - that means while saying something first goes subject, then object and then verb) while Japanese and Korean are SVO languages (Subject + verb + Object - first subject, then verb and then object).