I try not to reject invitations, but just ignore them. I think many of the people I've invited to by my friend do the same thing. The reason I don't confirm/reject friend requests is because the system for confirming friends is too slooooow. I now have 630 friend requests; therefore, to confirm or deny all of those people would be at a minimum 1240 clicks.
When people ignore or reject me, I'm curious why. Do they have the same problem I do, the system is too slow, do they have too many friends, have they not logged on, what else? Sometimes, I'm annoyed, particularly if I need that person's help with something.