That would be understood. I would probably say - "What size shirt do you wear?" Your phrase would be improved by dropping the "of" - "What size shirts do you usually wear?"

large : ) to be little wide :D

i usually wear M