Most Chinese I have spoken to have had much better English than they claimed / thought they had. There's a huge difference between being able to be understood, and having perfect diction. Yet in most cases, only the first really matters. Also - however bad your English is, I think that most foreigners' Chinese is much worse! I think that the difference is that unlike people like Europeans that come into contact much more regularly, Chinese are a lot more isolated, and less used to speaking to speakers of other languages, and perhaps also more used to thinking of language as an academic subject, rather than as a means of communication. So, they worry about it, rather than just 'having a go' and just trying to talk.

A lot depends on your individual situation and personality, but here's a quick suggestion that might work well in your case: Think about how much better your English is than their Chinese. I'm sure that will make you feel more confident instantly.