Some nouns work like an ing form. I think so it is this case.

Troll has dictionary meanings of to fish and to search. “Fishing for compliments” is a common phrase. Presumably, the speaker/writer means “fishing for tips” (encouraging the customers to leave tips (money).