I think the hardest language to learn will depend on what your native language is.
If you're an English speaker, I think Spanish and French are probably the easiest.
If you're a Chinese speaker, I think you might have an advantage learning Japanese because of the overlap in characters (kanji).
In general (for english speakers) -- according to a recent report from Ohio on language learning in the US:
Group I - French, Italian, Portuguese, Spanish -- 720 hours
Group II - Farsi (Persian), German, Greek, Hindi -- 1320 hours
Group III - Polish, Russian, Turkish, Vietnamese -- 1320 hours *
Group IV - Arabic, Chinese, Japanese, Korean -- 2400 - 2760 hours
A typical four-year high school in the US will only have 480 hours total.