The mastery of any language, including English, involves the practising of both receptive skills (e.g. listening and reading) as well as productive skills (e.g. writing and speaking). Productive skills may be what you are lacking. For non-native learners, speaking fluently may be a daunting task as it involves listening, interpreting what the other person means and coping with expressions that may be unique to the target language. When you throw in the fact that speaking is a real-time activity (unlike writing which can be completed in stages), it can be very daunting indeed. I am not saying that all learners will be better in writing than speaking (I have seen otherwise). What I am saying is that having a regular partner to practise speaking on a variety of topics will be more effective over the long run because listening to podcasts trains you on receptive skills only, and you miss out what you actually need most -- productive skills.