Native speakers may not be good judges of how difficult their own language is. Monolingual native English speakers (who mostly don’t interact with multilingual people) tend to think English is difficult to learn. Most learners of English don’t agree.
As someone who only just started learning Chinese, who is still at the pre-beginner level, I think that there are aspects that are more difficult (distinguishing tones in listening and especially speaking, using pinyin as a pronunciation guide), medium difficult (learning about radicals and identifying different characters), and not difficult once you get used to it (grammar and sentence structure).
Understanding each character represents a <em>syllable (including tone) </em>was what allowed me to get the basic hang of things.