We say "to my surprise", "to my disappointment" and "to my shame" to show how something affected us. "To my <noun>" = "It <verb> me that" "To my surprise" = "It surprised me that..." "To my disappointment" = "It disappointed me that..." 'Joy' doesn't work as well, but you could say "It gave me joy that..." 'Anger' and 'gladness' could fit this way, but these phrases are just never used. Not because it would break any rules, but just because the phrases never came into use. Also, "To my...." is not heard very often in normal speech any more, even though people would understand what you meant.