What you've said is fine. As Karma says you can do better than "that day" to replace "yesterday", although personally I don't think you can use "yesterday" as that would refer to the yesterday of the main clause (i.e. the day before this sentence is being said, not the day before Mike's walk). You could say
...the day before
...the previous day (although I don't think this works so well in this context - I'm not sure why not)
The double use of "had" here is very correct, especially for writing. An alternative is to replace the second verb with a simple past:
Mike said that he had been walking in the park when he saw him the day before.
This is less "correct" in a strict sense but is often used, as too many "had"s can feel rather heavy. (It is a recognised construction, by the way, not just sloppy English!)