Masahide, I think that the potential negative impact of living in a place where your target language is not spoken depends on the stage at which you find yourself. Your English is already advanced and you benefitted from living in a place where English is the primary language for several years. Maintenance requires less time than the early stages of acquisition.
Here is a post about maintaining one's level in a language from a writer who has some original and rather blunt opinions about polyglot aspirations, language, language acquisition, and time:
They suggest that half an hour a day, every day, is enough to maintain one's level. It is more of an effort to make time for and seek out resources in the target language when you are not surrounded by it, but it isn't difficult once it's a habit.