Coldness is an uncountable noun and refers to the state of being cold in a general sense. The sentence could be written as "He discerned coldness in their welcome." which would describe a general coldness he received from them. "A certain coldness" describes a specific type of coldness, which I suppose is why the article is needed. "A certain coldness" is nonetheless unspecified and vague. Your question is a good one and in the end I think this is just one of the many expressions that can defy grammatical analysis. I hope that helps.