Instead should be one word ("in stead" would have a related, but different meaning) but yes, it would be acceptable to say "I went to borrow a book instead of to buy one". However you probably wouldn't hear it normally expressed that way; it would be more likely to be said as "I went to borrow a book instead of buying one." Note that both "buying" and "doing" end in the letters "ing". These are called "gerunds"; that is, "A verb form which functions as a noun". The key point is that while you have most COMMONLY seen "instead of doing", that is only one possible word that "instead of" can come before. Most commonly (but not exclusively) it will come before a gerund: "I borrowed a book instead of buyING one." "I walked instead of RunnING" "I relaxed instead of doING my work."

The phrase "instead of to do" is understandable but clearly wrong and unnatural. Don't accept it. The simple rule is this: the verb form that follows "of"(=a preposition) is always a gerund.

@Bill: "Sth" = "Something", when it doesn't matter what the specific something is. In Italian dictionaries you would see "qc" which is a shorthand for "qualcosa"; same principle.

"sth" ? ---> "this" my guess