The Foreign Service Institute considers the following languages the most difficult for native English speakers: Arabic, Cantonese, Mandarin, Korean and Japanese.

http://www.effectivelanguagelearning.com/language-guide/language-difficulty

The four 'main' difficult languages for English speakers that you always see mentioned (in terms of taking the longest time to reach proficiency) are Mandarin, Arabic, Japanese and Korean. 

I think Polish, because I've never met any foreigner coming from Western Europe who spoke with perfect accent. And it's difficulty comes not only from hard pronunciation but also because there are no strict rules when making sentences. This may be considered as easier to speak but very hard to listen and understand the actual meaning for example, we have two sentences:

Ta się plama nie wypierze.

and:

Nie wypierze się ta plama.

which both means: This spot will never get washed. But of course they may draw attention to other problem.

And this is not the ending of your problems, we have also conjugation of verbs with respect to 2 sexes and neutral and also plural sexes. Declination of adjectives and nouns including 7 cases used commonly in everyday language.

The difficulty is that you have to extrapolate ideas of Polish language to a much simpler language like English. Unlike in reverse direction we have in Polish almost all equivalent grammar constructions to English incl. 16 or 17 tenses. Even though we have actually only four.

I have much respect and really appreciate everyone who tries to learn Polish. Try it hard and never give up!

Greetings from Poland,

Maciej