For me the hardest thing is just learning how to speak with confidence. I can read quite well, and I very much enjoy the writing system, but I get scared when I have to speak it and I always make silly mistakes. 

I think for the numbers, it's just a matter of memorizing which ones to use. "Ichi" is from Chinese, and is used most often. "Hi" is the native Japanese sound, and it's mostly only used in words like hitori (one person) and hitotsu (one thing). Tsuitachi I think is just an exception. I'm not really sure where it came from. I had a hard time learning the days of the month (tsuitachi, hutsuka, mikka, etc.), it just took some memorization. 

This probably sounds like bad advice, but just skip numbers for now. There's an entire section devoted to them in my grammar book, and I just don't think it's on the top of your priorities in learning Japanese. Instead, focus on things like stem, te-form, desu masu chou, causative passive, volitional form, etc. 

It's time much more well spent. :)

Besides, you can just check the pronunciations if you're unsure. After a while, you'll remember, don't fret over them. 

Ahhhh numbers! … I learnt basic numbers without realising there were counters or variations. When I found out numbers were a little more complicated, I threw them in the too hard basket (for later learning). At the moment I just use the basic numbers and hope people are tolerant when I get it wrong.

Grammar and vocabulary are pretty hard for me. I get frustrated when I can't say something the way I want to or express myself clearly, just because I don't know the right word or way to say it. This is also an obstacle for me when I'm listening to a song, t.v. show, etc. in Japanese. 

Japanese desperately needs to improve its writing system. For now, there is nothing closer to chaos