You probably need something else before 'memory', like 'my' or 'your' or 'his', etc. It could also be '...dig into one's memory' for something unspecific.

Its actually a good phrase dig into memory..to recall the memory what we have did

I am trying to remember is a better way to say it.