Couldn't is the better choice. If you were to choose "can't," it would be gramatically incorrect.

I'm not sure if this is a textbook example or a personal example, but I would even tweak the second response a bit: 

--Mr Smith must have been to your home this morning.
---No, he couldn't have, because he didn't know my address.

I would probably say: He couldn't have been, but couldn't have is a more colloquial way of saying it.