If you are talking to someone in your own language, you can give just 80% of your attention and still understand what they are saying.
This is because anything you miss, you can fill in in your mind from experience with the language. You can guess what will come next.
If you are listening to someone in another language, it not only takes all of your attention, but you must work very hard to keep up. Even a little distraction for a fraction of a second could make you lose the string of thought that you have put together in translating what the person was saying.
At least, that's MY theory.