As you said, 'loudly' is an adverb, it modifies the meaning of a verb. So if you said 'he listens to music loudly', the loudly would be modifying the verb, which is 'listens', in other words you would not be saying that he listens to loud music, but that he made a noise while listening to music, perhaps by singing along, or banging a table in time to the music.