Robin, you could also hire an italki tutor for a few lessons, more as practice rather than for learning. That way, you practice conversation, and you have a person who will talk with your for a whole hour, in your target language and falling back to English if you need to. You will observe how the teacher fills in awkward gaps, by talking about/using vocabulary you know and making the best conversation from that. You will gain confidence. You will gain confidence to meet free partners and fill in time making the most of vocabulary you have. And perhaps you can be like me, and take a convoluted detour around words I can't yet translate. All fun.

My country has an idiom 'อยากได้ลูกเสือต้องเข้าถ้ำเสือ it directly means If you need a baby tiger, go ahead to where it is without any fear 
By the way, I'm Chinese mandarin learner as well. I'm at the workplace where there are many Chinese and the biggest problem is that they don't even speak a bit English. In order to communicate with them I need to improve my Chinese. At first I no nothing about Chinese then I keep improving. Now Chinese always my Chinese pronunciation is very good, but actually I can speak a bit. All I want to tell you is just don't be afraid to do what ever you want. My English grammar might Sounds weird for English native speaker like you. Hope you don't mind 

Do  not  keep  silence,  especially  if  you  are  not  good in  the language. Ask questions to which  you  know  answers if  they are  not related to grammar. Learn simple descriptions if you cannot pick up a precise word. It  will look  stupid,  but there won't be the freaking silence. I know , it's easier  to say  than to make.  And  with some people all  these  tips  do not work, but in general  they do. :D