Cleary, you've gotten to the diminishing returns stage. I'm an English learner who has been learning at school in Boston since July last year. I don't know what English level I'm at right now, but I personally think that once you get to that point, all you can do is to keep trying to learn what you don't know from various types of things such as youtube, movie, books and so on. As long as you can learn something from them, you're obviously improving. I understand that casual forms are the one of the hardest. If you already have some Japanese friends to practice with, just keep at it and try to copy expressions they use in speaking. As well, you could also learn more common Kanji and set your goal to understand more than about 1000; as you probably already know, 500 kanji is definitely not the enough number to read a book. Reading is very helpful. Also, one of my strategies is to be as flexible as possible and never think "my skill is Ok or enough". If you don't think your Japanese pronunciation or intonation is not as natural as Japanese speakers, you can learn from movies, TV shows or TV series. You should never compare your skills to another Japanese learner's but Native Japanese speaker's. As a final advice, always having something you don't understand in your head and keep learning until you figure it out. Once you've gotten to the stage, you can improve only little by little, but I believe it's not uncommon and the process of learning another language. Hope this is helpful.

Like Yoshinori said, you should always challenge yourself! :D I try to say what I think in Japanese all the time and when I do that, I will always end up learning new words, since I need that in order to express my thoughts :D Manga is also an extremely good source for learning both kanji, slang and advanced grammar. I actually learnt a lot of JLPT 2 and 1 grammar for deathnote and kanji as well :) Good luck :D