Both indicate that the event occurred in the past, occurred over a period of time, and has ended. They are usually used in relation to some other event; either in answer to a question or with an additional clause.
Both are used commonly in English, and both are grammatically correct.
As they are used in English, they are interchangeable. The past perfect continuous (had been studying) places slightly more emphasis on the continuous and past nature of the act, but only slightly. Context and tone would matter more for emphasis.
I am trying to think of an instance where they cannot be used interchangeably and cannot think of one.