I love this expresion! You shouldn't think of using it or not according to what Grammar says, but according to what French people say and in which context. This expression is totally correct if you're in an informal context. You can hear it from people who speak with you, in movies, etc. So it's good to know it. Of course, don't use it in a formal context or in an exam!