In this case, if you say "I study history at school," you are most likely telling someone where you study.* Example: "Where is the best place for you to study?" "Well, I can study math anywhere, but I study history at school because I never have the motivation to do it at home."
And when you say "I study history in school," you are answering a question more about your field of study. Example: "Wow, you really know a lot about the middle ages!" "Yeah, well I study history in school."---so for this one, because you are "in school" (inside the time of your life where you are furthering your education by schooling) "in" is appropriate, not because you are "in"-side the school studying. You would have to say "I study history in the school (building)." which is weird and much better to say "I study history at school."
*That all being said, "I study history at school," could also be used to answer a question about your field of study. "What are you studying?" "I study history at/in school but I study music in my free time."
here's a helpful site I found online. Of course there are exceptions, but it's useful.
http://www.learnenglish.de/mistakes/atvsin.html