Anyone, Am I the only one that struggle to listen to someone who is talking english?

Hey. I am an Indonesian. My native language is Indonesian and Javanese language. I only learn english from the school since I was 1st grader. I've never take any english course before. Outside the school, I used to learn english by reading some english articles or english contents on the internet, watching english movies with subtilte on it, listening to english songs. It makes me easier to improve my skills in reading, writing, and speaking.

But my listening skill is not good. I can listen to someone when they speak slowly or moderately and clearly. Indian and British accent are the most clear accent for me. But, when I listen someone speaking in english fastly, I can not understand what they mean. Sometimes many people, especially english native speaker speaks too fast and hard to understand.

And..... here's the complicated situation;

Even tho I can't understand the english speaker who speaks to fast, I still can listen to Indonesian people's english speaking well, whether they speak too fast, or not. I still can understand them easily.

What do you think, guys? Does anyone has the same situation with me? In your opinion, what is the cause of my problem? Or maybe you have a solution?