Both meanings apply. The grammatical meaning of that structure of words is just that, in general, people say that he is 108 years old. However, just understand that it just means "he is 108 years old." This structure implies that even though people say he is 108 years old, he might not actually be 108 years old. The person speaking doesn't know for sure.

I can see that you're still studying passive constructions. This is another variation. The explanation in your book is correct.We don't know for sure how old the man is, but people say that he is 108. Active: People say he is 108 years old. Passive: He is said to be 108 years old. Active: People think these pyramids are the oldest in the world. Passive: These pyramids are thought to be the oldest in the world.