I understand what you would like to do. You would like to find some underlying logic to grammatical rules.
English grammar has not grown out of a regimented and systemic laboratory and so there is not necessarily any logic to grammatical rules.
Regarding "to", some people have rationalised its use by thinking of "the direction of flow" - from A to B. I went from Paris to London. Or "initiator of action and the receiving end", such as I talk to him. It does not always work. I hit him. Why don't we say "I hit to him"? I lick him. Why don't we say "I lick to him" or "My licks go to him" or "My licks are made to him"?
So you see? It's fun to try to find the underlying logic. There is often none, but then to many people half the fun is in being disappointed time after time.