The expression "to be worried" is very common. It has the same meaning as "to worry". In it, "worried" is an adjective (though it has the form of a past participle). So the only verb is "to be", which in the 3rd conditional, becomes "would have been". Here's another example:
"If West Ham had beaten Manchester City at the weekend, I would have been delighted."