In my opinion, it would have to be writing and reading. That requires the most time to master. Speaking takes some time to get used to when you are first practicing the pronunciation of all possible sounds and tones, but once you've finished that, it becomes second nature. Surprisingly, the grammar is also fairly simple. I've only ever had difficulty with remembering how to read and write Chinese characters.