Both indicate that the event occurred in the past, occurred over a period of time, and has ended. They are usually used in relation to some other event; either in answer to a question or with an additional clause. Both are used commonly in English, and both are grammatically correct. As they are used in English, they are interchangeable. The past perfect continuous (had been studying) places slightly more emphasis on the continuous and past nature of the act, but only slightly. Context and tone would matter more for emphasis. I am trying to think of an instance where they cannot be used interchangeably and cannot think of one.

the past progressive or the past perfect simple expresses the idea that an action happened before another in the past for exple he _had studied_ english before he moved to new york . but the past perfect continuous can be used to show that something started in the past and continued up till another in the past

We don't say "I was studying for 3 hours". "I studied for 3 hours" is better. "I had been studying for 3 hours" means that when a certain event occurred in the past, I had just completed 3 hours of studying. "When you called me last night, I had been studying for three hours."