If you don't do any listening or speaking, it's going to hurt you later when you need to use the language. When some ESL students come to California, they have problems understanding what people are saying to them (we talk fast), asking questions, and giving simple answers. Because they haven't practiced speaking, it's very difficult to understand what they are saying because of their poor pronunciation.