I was studying is correct if you are taking about the past (past continuous tense), but if you are using a conditional (second conditional) in the if clause or Wish, you can use both "if I was studying... or if I were studying..." or "I wish I was studying... or I wish I were studying..."