You can survive without it, by using the past simple.  But it's often used naturally by native speakers when fondly calling to mind the past.  Advanced speakers who don't use it would be missing out on quite an important means of expression.  

I would say that it's fairly common though we don't really realise we are using it because we would becomes we'd and I would becomes I'd.