'Spend on' is very commonly used with purchases, the object either being the item bought or the recipient of items bought for them. This is obviously what is happening here - 'on' makes it clear that the purpose of the spend, and the focus of the statement, is the quantity of cash exchanged for the pencil. The use of 'in' with a gerund is quite general, and can be used to describe a characteristic or aspect of the action: I saw no purpose in buying the pencil; in buying the pencil I became a pencil-owner. It is possible to use it here, but it seems slightly strange.

The more natural way to say it is “I spent $3 on the pencil.” But if you use “buying,” I think “in buying” is better than “on buying.”

Is that pencil made of gold ?