The answer depends on what you are trying to say. If you 'tried knocking on the back door', the implication is that you were attempting to get someone to answer (by knocking on the back door). On the other hand, if you 'tried to knock on the back door', then all you are doing is attempting to knock on a door, regardless of whether or not it would atteact anyone's attention! In fact, if you keep it as a single sentence, then you don't have to repeat the knocking verb at all - I reckon you could simply say: "I knocked on the front door, but no-one answered, so I tried the back door". It is very slightly ambiguous, but avoids using knock twice.

I knocked on the front door,but no one answered.So i tried (to knock) on the back door.