"set off" here means it showed it off. The dress "setting off" the waist means that the waist is really being complimented by the dress flattering her form. In other words, the dress fits her so well, it shows how thin her waist is, and what a desirable figure she has.