Hmm, although I am tempted to say: "kanji, Kanji and KANJI!", I learned to appreciate what kanji does for japanese.

Since there are only a few sounds possible and limited combinations, japanese has many words that sound pretty much the same (Im not even considering que ones that are pronounced slightly differently). The only way to know what the meaning is is through the context or Kanji . A text written only in kana would be pretty much indecipherable, due to the lack of spaces between words. So, although Kanji is hard, I love them and i am having a blast learning their meaning and seeing how they are put together.

One thing that is hard for us Gaijin to understand is the speech itself, how is constructed to be vague and not offend the other person. You are not to pull the attention to yourself during conversation, so the word "I" is not used as frequently as in English or Portuguese. And that is a hard concept do grasp, since it's so cultural.

I think it's difficult to distinguish the most challenging part of learning a certain language, as every part of it is challenging in its own way. I was about to say that kanji is the hardest part, but when thinking about it, I realized that to me kanji is more time consuming than necessarily difficult. Kanji is super hard in the beginning, and it does take a LOT of time to recognize, read and write kanji (especially the latter), but as I know more and more kanji, learning a new kanji becomes less difficult. It makes it a lot easier when you can remember it by the radicals that it consists of or by the kanji that it reminds of - e.g. 親 can be remembered as 立 above 木 beside 見, or as 新 but with 見 instead of the middle part of 近.
I do think that the use of kanji can be really difficult sometimes though, especially when you have two kanjis that almost mean the same, for example 見る and 観る.

Elementary Japanese grammar is easy, at least as soon as you've accepted that you just have to learn everything from scratch (no "direct translation" here!). When advancing to intermediate Japanese, I think the grammar becomes much more complex. You need to be able to distinguish between some grammar points that almost have the same meaning, yet the use of them or the connotation of them are completely different. Of course the intermediate level is always much more difficult than the elementary level of a language, but the difference between elementary and intermediate Japanese is much bigger than what I experienced when studying English and French.

One part of Japanese that's especially been confusing me lately is how many words for the same thing the language has - e.g. 両親、親、父母、父兄 or the difference between words such as 隅 and 角. The same is the case with keigo - I think that's hard to master! Especially when the words change completely.... but this diversity is also one of the points that makes Japanese so special and fascinating! :)

I'll do my list from hardest to easiest... Just a few things on a spectrum...

#1: Kanji! - Kanji is brutal (at least I am not learning Chinese hehe). The people that grew up in China had to learn like 1 billion characters are insane! I could never do that. At least in Japanese there are two other writing systems that are much easier and clearer. Kanji is also extremely annoying when learning both the onyomi and kunyomi! I wish the Chinese never went to Japan :(... The writing looks so much better with just kana! XD... But yeah, kanji sucks.

#2: Grammar - Besides kanji, the only thing that really strikes me as 'difficult' is the grammar. The grammar is somewhat different from English (SOV vs. SVO) and can confuse learners a lot... I also get very paranoid when writing Japanese sentences because even though the object goes in the middle of the sentence, I do not know which order the object goes in... Even though there usually is no specific order.

#3: Honorific speech - This is not a major difficulty, but speaking honorifically really makes me self-conscious when trying to speak Japanese! Even if it's a friend or my dad, I still feel restricted from saying things!

#4: Pronunciation - Japanese pronunciation is much easier in my opinion than many other languages. Some things can be a bit hard like: obasan vs obaasan. However, the language is usually very straightforward in terms of pronunciation. 

#5: Kana - At the bottom of the list, I am putting kana. I love hiragana and katakana! It's my fav! I think they are fun to learn, especially trying to remember them like the English language. However, this will happen over time, and is very hard to do! 

Good luck teaching! Just a perspective from someone learning it. 

I think the hardest part of the japanese language is definitely rememberring how to write kanji.

Rememberring the meanings and how to pronounce them isn't that much of a problem to me. (I quickly learn how to speak languages, speaking is so much easier to me than learning how to write)

But knowing how to write certain words is really hard. Also learning the correct grammar forms is pretty hard. ((To distinguish formal and casual japanese))

It would definitely be learning and memorizing kanji. How to say (in context) and properly write kanji is really difficult for most students learning Japanese.