I'm really curious, how do you estimate the number of words that you know?

Here is my learning thory about listening proficiency.

Basically, audio and graphic data are slightly different. Audio means listening and graphic means reading in rough speaking. They take different routes loading into your brain. Since you claim you have had profound capacity of English vacabulary, I consider you are capable of interpreting English words in graphics, standing for "reading comprehension". That is very distinctive from "listening comprehension".

Such a problem consists in incompatible learning in listening and reading which is often encountered in some of the countries where English is not frequently spoken. Therefore you had better modify  your learning as soon as possible or you will lose a plenty of time in futile tasks. In other words, you do not establish enough "audial information" in your database of brain to be retrieved while you are listening, as a result, even you have recognized ten of thousand English words, it is very likely most of these words learned with efforts are not formatted in the pattern of audio to be searched and recognized.  

There are so many factors to consider. 

Different accents, intonation, slang... I know Brtitish people who have troubles following to american movies. So yes...

I am Swiss and therefore I speak a Swiss dialect. Even other Swiss people can't understand me well because of my strong accent... Switzerland is very small...

Don't be too hard to yourself!

@Priscilla:

You had some very good points!

Being fluent means for me that I can communicate with another person over a longer  period of time and  feeling very good about it. 

I