On my goal to becoming a polyglot, I have suffered one major pitfall: Thinking in another language.
For example, when I am watching a T.V. show in another language, or playing video games, I'm able to distinguish key words and phrases that I've already studied. When I'm focused on my studies without distraction and using textbook examples (or other resources) to help provide me with content - I can easily generate what I think to be the appropriate response. However, if I go to a restaurant trying to order something in a foreign language... I find myself struggling to think. Not because I'm nervous or lack the vocabulary - my brain just slows down and I end up speaking English so that the person I'm talking to doesn't get impatient or irritated. Is part of the problem due to the fact that I'm becoming distracted from my perception of the listeners expectation? It seems likely, although when I am trying to think of sentences in my head, and no one is around, I'm still slow! Perhaps I am projecting unnecessary expectations on myself, too. Anyhow, I digress.
Other than living abroad where native speakers reside, what have you done, and what is helping you to think quickly in the languages that you want to learn?
Just a couple of thoughts...
In your examples where you do understand and manage well, you have some form of guidance. When you don't have that kind of security, that's when your second-language skills fall apart.
So you need some self-guidance, and that's where the grammar and phrases - not only the vocabulary - come in. Just accept the Japanese grammar (for example) as a necessity, and practise fitting your ideas into the structures of Japanese sentences. Treat it as a mental game! :D It won't be easy at first, but the more you return to "English thinking", the more difficult it will become.
Another thought is training your ability to react. Role-play exercises are very useful for this... or even better, working with a teacher or language partner, only using your target language.