It means the normal definition of try. When someone says something like, "You think [blank]? Try [blank]!" They're challenging your belief of something. "You think my teacher is nice? Try talking to her before she's had her coffee." or "You think your math is hard? Try Calculus!" They are challenging you to try to do something that they think would prove you wrong.

If you were to do this, your situation would be even worse.

Hello, In the sentence, "try" means make an attempt or effort to mess with ........ I hope this answers your question. If you’d like to practice speaking. I’d be happy to help you develop a study plan. Feel free to view my teaching profile and class offerings and send me a message if you think I can help you. Regards.