Hi Jesús, to be at one's wits' end means to be so worried, confused, or upset that you don't know what to do next. In your example sentence, the person has looked everywhere for his or her dog, and is now feeling bad and doesn't know what to do.
e.g. I can't sleep because my neighbors play loud music every night until 3 am. I am at my wits' end.