The first sentence below is correct. If you add "one" to the second, it would be correct as "My pencil is sharper than that ONE on the desk." Even without changing it, it might be considered correct.
"My pencil is sharper than the one on the desk."
"My pencil is sharper than that on the desk."
These two sentences are both correct and are interchangeable.
"My pencils are sharper than the ones on the desk."
"My pencils are sharper than those on the desk."