I'm pretty sure Morrissey is singing about the wrinkles on the face of the person he's addressing in the song (the one he's wishing an unhappy birthday). It's literally about the person's wrinkles, which correspond to birthdays and aging. This isn't a typical phrase. We will, however, describe a face as sagging or saggy or, alternatively, droopy. As you mentioned, it's also figurative in this sense as a sagging face often looks like a sad face.