The point is what you mentioned
You need to understand while they speak fast
In order to achive that you can start by cartoons which are less complicated , download an English subtitles but at first try to write down the dialogues while you hear( without using subtitle), you may use pauses many times to understand a short conversation but as you proceed you will be able to write a longer phrase with less stops.
Then after a five minutes of writhing check your self with the subtitle to find out your mistakes
I am sure after the second movie with above procedures you will feel the improvement in your listening