I believe that understanding what these native speakers say is not obtained over night. Practice and practice and practice. There's always room for progress. And then it'll be fruitful!

it's a long progress. I listen to BBC from time to time, it helps.

If it is difficult I would first try with English subtitles, not subtitles in your own language. The more you watch movies and series in English the more you will start to get used to the accent. Reading will also help, because it will help you to become immersed in the language, and will also improve your vocabulary. Definitely stick with it and it will eventually come. Sherlock is difficult because I find that Sherlock himself speaks very fast. Try some other series, especially American ones, to build up your confidence again and it will definitely come eventually.