I'm trying to think, but I can't come up with any logical reason as to why one is correct and the other isn't. All I feel when I hear "listen music" is that it's wrong. I think that's just the way it is. English is full of contradictions and exceptions. On a side note, I think you should say "English speakers", instead of "Englishmen". Englishmen means specifically someone from England, but there's a lot of other places where people speak English!

I think you've misunderstood a very basic part of English. "Listen" is an intransitive verb. "I listened" is a complete sentence. If you want to add an object then you use a preposition. If you say "I listen music", then I understand that it's not correct and I wonder if you don't know or don't care. "Watch" is intransitive when there is no object, or the next preposition is "for", which then means the action of watching is in anticipation or a search for something. If you add an object (movies) then "watch" works as an intransitive verb. "I watch to movies" is simply incorrect. Another red flag for "don't know/don't care". That's how it works technically. Alternatively, you can just accept that different verbs work in different ways and just learn them. This has nothing to do with passive behaviour.