Hi Wendy,
It would be:
¿Por qué no visitaste a tus abuelos? OR ¿ Por qué no los visitaste?
'Les' is an indirect object pronoun. 'Los' is a direct object pronoun, and it's used to refer to something that has been mentioned already.
Les escribiste una carta a tus abuelos. This letter was written TO your grandparents.
You say 'a tus abuelos' to identify who the 'les' is talking about.
No sé dónde están los libros. Los perdí ayer. The books are lost. The books are not TO/FOR anything/anyone.
I hope this helps.