Hello Rui Ding! As obvious as the answer may sound, you simply have to talk to and listen to people more often. YouTube has some great listening exercises on language-learning channels that you can use to improve your listening. As for speaking, that's what this site is for.
If you can find a language partner willing to speak with you regularly, that's great. If not, contact a teacher or tutor and set up some sessions where you can have conversations with them. The more you practice, the better you'll become =)