This one is tricky! I think it kind of depends upon whether we reference the present and how long ago the past feels, while we're telling a story set in the past. So...
"Yesterday I went to the library, which is the biggest building in town, and met my friend."
[If we say "was" here, it does sound odd because by using "yesterday" we're putting this past in reference to the present. And also, because yesterday is very recent, we won't expect thing will have changed since then.
If we start that sentence with "20 years ago"... we still reference the present, but now it's likely that things have changed in that time, so it feels natural to use "was". This doesn't necessarily mean that it's no longer the biggest building. But if we want to be clear that things have changed, we could include "at the time." Or we could say, "was and still is..." to indicate that things haven't changed.
In a story set entirely in the past with no reference to the present, we'd normally (but not necessarily) use "was".
"In 2015, I went to the library, which was the biggest building in town, and met my friend."
[Using "is" not only relates the story to the present, but also to reality. So if we used "is" in a fictional story in this case, it would make the story seem somehow more real, or at least it would seem that the narrator believes it to be real.]
As I said, tricky!