"this" refers to an object or topic that is close to the speaker. Either close in physical position or close in the sense of time.
If a person points to an item in front of them and says "this is mine." They say "this," because the object is physically close to them. "It" can also work in this context, but consider the usage of "it" or "this" in the following conversation.
A: "Someone left a bag in the other room. Do you know whose it is?"
B: "Oh, it's mine. Thank you for telling me."
In this conversation, ONLY "it" is appropriate, because the object is far away from the speaker ("in the other room"). If they said "this is mine," that would sound quite strange.
In the example you gave:
I can do it.
I can do this.
You can use either "it" or "this," depending on the context. Based on the phrasing, I assume the "it/this" refers to a task.
If the speaker is thinking about a task that will happen in the future, then "it" would be the only option in that case. For example:
"Tomorrow, I have to run a marathon. I can do it. I am confident in myself."
Since the "marathon" that "it" refers back to is not "close" in time to the speaker, "it" is the only option that will work.
However, if the speaker is standing on the track just before the marathon and he thinks to himself, "I can do this," then that would be fine. The reason is that the marathon is temporally "right in front of him." The task is at hand in that moment.
Does this make sense? If you would like more examples to make it clearer, please let me know.