It depends when and where you begin. For babies, all languages are easy. :-)
For adults and teenagers from an English (or sometimes Spanish) language background, the US Defense Language Institute divides languages into four categories (https://www.ausa.org/articles/dlis-language-guidelines
). The hardest category includes Arabic, Korean, Japanese, Mandarin (Chinese), and Pashto. I've no doubt that Cantonese, Hokkien, and all other Chinese languages would also belong in that group. I understand that Burmese is also very difficult (it has tones like Mandarin, politeness changes like Korean, and its own alphabet).
But obviously, this all depends where you begin. When I was studying Mandarin, I had a classmate who was fluent in spoken Cantonese (and could read some Chinese characters). She learned much, much more quickly than everybody else. Every time the teacher asked "do you understand?", my Cantonese-speaking neighbour would say "tingdong!" ("yes!"), and the teacher would start the next point. ;_; ;_; ;_;
And there are also some languages where there are very few textbooks and other learning materials. A friend of mine is learning the Ashe language from Nigeria for his research work. There's no Ashe-English dictionary. In fact, there are no Ashe dictionaries at all! So that is a very difficult language for everybody, unless you are very lucky and already speak a nearby (Plateau) language.