Albeit the very small difference, both sentences have vastly different meanings: concern vs regret
어제 그 책을 샀어야 하는데
You still don't know if a particular someone bought the book yesterday.
You are worried that he/she might have not bought the book, and express your concern.
"I just hope he bought the book yesterday"
어제 그 책을 샀어야 했는데
You know that nobody (including yourself) didn't buy the book yesterday, and express regret on this.
"I should have bought the book yesterday"
============================================
Another example:
걘 지금쯤 도착했어야 하는데
He must have arrived by now. (you are worried about him)
걘 지금쯤 도착했어야 했는데
He should have arrived by now. (He didn't arrive on time , and there will be consequences)