This is interesting. I think you are right that the sentence is actually ambiguous. Nevertheless, it would always be understood to mean that the prime minister was telling light, amusing stories about various events in his life, in the old days. Without any other context, "the reminiscences of Dan" means that Dan is reminiscing. For "reminiscences of X" to mean stories about the old days on the topic of X, you would need a clear indication of the person who is reminiscing. If X is another person, that would need to be a very clear indication. Finally, although you could use the word "reminiscences of X" to mean somebody's stories about X, it's not a very likely choice. "Reminiscences about X," "anecdotes of X," "stories about X" would be more likely.

"Reminiscing" is the act of thinking or talking about past experiences. Since the Prime Minister isn't the speaker in this sentence, it suggests that he was talking about past experiences. Since the sentence refers to his "reminiscences" as amusing, the speaker found them funny.