I've never heard anyone refer to the police as "the brass". There is, however, a phrase "the top brass" which means "the people in authority" within an organization, so I would guess it means that she reported him to her bosses. "To set the dog on someone" means to order or let your dog loose so that it chases someone down and attacks them. So, "set the brass on someone" is just a play off of that phrase. In the example you gave it would mean that she spoke unfavourably of him to "the brass" so that they had him in their crosshairs. For example, you could threaten a trespasser by saying "Get off my property now, before I set the dogs on you". Para que quede claro: Ella hizo que "el brass" lo investigara y buscara una razón para perjudicarlo.

I agree with Spangola. It's not a normal expression using BRASS. It would make more sense if the military was involved, as BRASS could refer to upper level military.