"You were studying" is the correct answer

Uiseong is correct. 'You were study' is not possible at all. The verb 'be' is followed by adjectives, nouns, -ing forms of verbs, past participles. It can never be followed by a simple form of a verb.