It seems like only yesterday that we first met (here). It seems like just yesterday that we first met (here). If you take away the "here" then you are not emphasizing that is was HERE that you met, so it that is important to your meaning then you should keep the word "here" in both cases. You don't need "it was".