"They didn't think he had been TO VISIT his birthplace since he WAS a small child." = Correct.
It is not a very good question. It may be possible to say
"They didn't think he had been to visit his birthplace since he HAD a small child."
But we need a little more context about the man/person who had the small child, or hadn't visited his birthplace since he was a small child".
Easiest way to work out these sentences is to forget grammar, and learn what words are past and ensure all words in the sentence are past.
Didn't = past, Had = past, so "had been" also = past and last part must be past ="was" or "had a small child".