べんきょう ( 勉強 ) is a so-called suru verb. When directly followed by suru, as in, べんきょうする、 it means to study. So, 日本語を勉強する (にほんごをべんきょうする) means study Japanese. Like most suru verbs, if you drop the 'する' then it becomes a noun. So べんきょう on its own means study - as in, the study of Japanese. If you follow a noun with をする then it means 'to do..(the noun). So you can tennisu wo suru - literally to do tennis, although of course, the more natural translation in this case is to play tennis. So getting back to your question, the suru in the sentence means that you 'do' study at the library. This could also be expressed as としょかんでべんきょうします。I hope that helps - I do vaguely remember finding this confusing when I first started learning Japanese!

In Japanese nouns can be combined with します to create verbs. For example, パーティーをします means to throw a party. Thus is similar, but i dont think that べんきょう is ever used on its own as a known. These types of verbs are called する verbs, and often times the を can be omitted.