This is a tricky question. Let's look at the verbs "say", "give", "tell", "speak", and "lend". To play it safe, you can't go wrong with any of them if you use this form:
I [say, give, tell, speak, lend] something to you.
However, the next form (without "to") only works with some of the verbs:
I [give, tell, lend] you something.
For the verbs, "say" and "speak" you need to insert "to" in order to make sure that "you" is not misinterpreted as a direct object:
I [say, speak] TO you these words.
All the examples so far have both indirect and direct objects. If only one object is given, it will be interpreted differently as indirect or direct for different verbs:
I say something. (direct object)
WRONG: I say you.
I give something. (direct object)
USUALLY WRONG: I give you (unless "you" are the thing being given. For example, an employer can give an employee to another business.)
I tell ______ (either direct or indirect)
"I tell Bill" (indirect) or "I tell a story" (direct)
"I tell you" (indirect)
I speak something (direct)
"I speak English"
WRONG: "I speak you". unless "you" are the thing being spoken, which is hard to imagine. However, with "it" it is natural:
"Do you speak English? Yes, I speak it."
I lend something (direct)
USUALLY WRONG: "I lend you" (unless "you" are the thing being lent. For example, an employer can lend an employee to another business.)