I think you should say it as it is. Say that you don't really feel you two have any interests or anything in common. You will probably make him feel dissapointed, yes, but not much, I believe. It seems like you two don't know each other very well, so the earlier you let him know that you don't want to speak with him, the better. He will be more hurt, if you pretend you want to talk with him for weeks, and then suddenly say you won't.
So, yes, let him know what you think now, and don't wait.