If you are talking to someone in your own language, you can give just 80% of your attention and still understand what they are saying. This is because anything you miss, you can fill in in your mind from experience with the language. You can guess what will come next. If you are listening to someone in another language, it not only takes all of your attention, but you must work very hard to keep up. Even a little distraction for a fraction of a second could make you lose the string of thought that you have put together in translating what the person was saying. At least, that's MY theory.

I think maybe because you feel confused??? haha. I think I would feel the same way if someone was speaking to me in another language.

Here is a suggestions for your question: Why am I always distracted by other things when someone's talking to me in English?

Simply because you're not paying proper attention to English... the way your own question is written already shows that.