Coming from a language that uses an alphabet rather than pictorial characters, it's obviously difficult to memorise Chinese characters. In addition, the tonal system and the sounds that aren't present in English, as well as trying to express myself well with a lack of verbal conjugation or tenses. 

In short - the most difficult part of any foreign language is the parts that your native language doesn't possess as you have to learn alien concepts and ideas and force your brain into thinking like that!

My difficulties to learn Pinyin and remembering characters.  Pronouncing,  too

Learning Chinese characters are not too difficult for Japanese!

Pronunciation can be difficult, but I won't get into Chinese so deeply, so it is OK.

Probably, for Japanese, the barrier between the countries are the biggest problem.

Alphabet and Pronunction :|

Tones and written characters.