My guess is that the accent can throw you off, if your experience in English listening comprehension is limited to the slow and deliberate pace of speech common in classes. 

It could be difficult enough that certain simple words can sound unintelligible to you, even if you have heard it many times before.

My advice is to simply listen to audio books for a start, where you may have access to the written text as it is read to you. There are other methods, but this works well for me.