I would add an 'at' at the end of the sentence.
So, "Do you have a telephone number or an address that I can reach them at?"
If you were just asking for the telephone number, I'd say "Do you have a telephone number that I can reach them on?"
Or just the address, it would be "Do you have an address that I can reach them at?"
Hope that helps