"Yes, I was." The "worried" is implied in the response, so it's the same as: "Yes, I was (worried)." It's like past perfective in Russian, because you were worried in the past (at that time), and it's a completed action.
A past imperfective answer would be like: "I had been worrying," i.e. during that period of time, I was in the process of worrying.