Possibly better to use the term "retirement home" as it sounds a little more respectful.

I planned going to old people's home to speak with them

If I understand your question correctly. The correct statement is. I plan to go to an elderly home and speak with the elderly.