The sentence is fine as it is. The first "there" is a "dummy subject", with no place information. The second "there" is an adverb of place. Removing the second "there" removes information about place, changing the meaning a little, but still leaves an OK sentence. Have a look at https://www.perfect-english-grammar.com/it-and-there.html

You can also write: As far as I can remember, there were not any buildings.