It's natural to be nervous in a situation where someone is trying to communicate you but you can't fully understand. You have to become comfortable with speaking and listening to a language before you can get over it. That includes increasing your vocabulary so that you can speak and understand about more topics, and also simply continuing to expose yourself to the language so that you grow more comfortable with it.
Most of all, don't let those feelings scare you off. ^^ Keep studying and keep practicing regularly and you will grow comfortable in time.