I agree with you. They differ in terms of usage in the sentence. For instance, an infinitive without to follows modal verbs while some specific verbs must be followed by to infinitive. e.g: You can have some tea. I'd like to have some coffee.

First off, the infinitive has no meaning aside from naming the action expressed by the verb. But it's unique in that it never refers to any specific person (=doesn't conjugate for person or tense), and can act like a noun: "To complain about the president would be considered treason in some countries." The only difference is in syntax: some grammatical forms require *to*: "I want/need/try *to* help." Other forms don't take it at all, like modal verbs: "I can, will, should –[to] go." And with some forms, it's optional: "This book will help (to) teach you English."

By the way, the infinitive without *to* is called the plain infinitive.