I think the main confusion is the phrase "by what authority". It basically means "for what reason" and stands for "why".
Also, "the writers of dictionaries and grammars" is the subject of "say". They say something. So, "to" cannot fit there. It would be wrong. "What they say" is the object.
Also, although "ask" usually needs a person who is asked, in this case it's a general question.
Few people ask [by what authority] [the writers of dictionaries and grammars] say [what they say].
Few (not many) people want to know why they say X.