Hi
Yes in this context "pull off" can be used. It doesn't really refer to if the clothes fit in a physical sense, but more that they suit your personality.
But I guess there is a relationship between physicality and things like confidence.
It can also be used for things that are not about clothes, but about appearance. For example "He tried to trick the school by pretending he was a teacher, but he couldn't pull it off." = "He tried to pretend he was a teacher but he was not convincing."
Be careful with "pull off". Like many phrasal verbs it has multiple meanings in different contexts.