I thank at the beginning listening and speaking are the hardest because of what you mentioned: different accents and the speed of talking. This is especially true with languages where words are joined together like French for example. Speaking is also hard at the beginning because you are unsure of yourself and afraid or embarrased to practice with native speakers. 

However, at a more advanced level writing is definitely very difficult because it requires great grammatical precision. Speaking is often more informal and relaxed, the important part is to get your point across but mistakes are often more evident in writing.

I agree with you!

Speaking is definitely the hardest part when learning a new language.

This is my second time learning a new language and in both cases speaking has been the most dificult.

One thing I found of much help is to listen to music and follow along with the lyrics, this helps you hear how the words are spoken and it also helps you with your reading comprehension.

Keep it up!

Personally, I find writing the hardest. I can manage orally and if you watch movies, listen to music, etc. you can pretty much improve your accent troubles and get accostumed to listen to certain phonetic patterns you were not very used to.

When it comes to speaking, you just utter words and copy others trying to get a hold on idioms and more generic sentences that people use everyday (:

Then again, it's in writing when I find the landscape a difficult one since you have to be very fluent so when you start toying with ideas for your written assignents you will think following the structures of the language you want to write in -and not thinking about the structures to your native/your second language. The disassociation can be very hard to overcome, but overtime it gets easier and you get more comfortable writing this way and not the other ^.^

Cheers, Victor!

Deffinately writing and probably listening. 

I agree with @Jasmina Uzunovic