I'd say it means "I thought about studying vocabulary, but I'd feel lost and helpless with it".
요리를 하자니 귀찮고, 나가서 사먹자니 그것도 귀찮다. I thought about cooking but I can't be bothered to cook, and I can't be bothered to go out for breakfast/lunch/dinner either.
-자니 is colloquial Korean and doesn't translate comfortably into English.