It's not much of an idiom. It's just a turn of phrase by the speaker. You can understand it to mean "it feels as if you're listening with the wind blowing directly at you", ie. you can only hear the sound of air moving.