All these skills require time to master them. 

Passive skills as reading and listening are easier to achieve, while active skills as speaking and writing are more difficult.

I think that the most difficult is writing. I mean writing on a good level. I think so because in writing you have to be perfect. If you do mistakes in speaking, most people don't pay attention to it, but every mistake in writing like spelling, grammar, punctuation, even style is noticeable. 

The most difficult skill is speaking. In fact, there is a good reason behind that which is the fact that you can not learn speaking from any resources like books and videos ... etc..

Moreover, you have to practice with someone who is either a native or a really fluent partner, for you to be corrected and directed through your learning process. Furthermore, I noticed that the most effective way to remember what you have learned (learnt) is get the credit from native speakers, that is, when you say something and it is undrestood from the first time you will never ever forget what you said and will not be at that same level before you say it.

All in all, speaking is the hardest skill due to the high level of participation it requires and the need for approval from a native speaker.