The origin of the phrase seems to go back to observations that elephants follow the same paths and even hand down genetic memories of directions and places grounds across generations. Each elephant clan has a certain burial place, like many human communities, and always help the dying ones get back there if they are not killed traumatically first.
The most famous example of the tenacity of memory of elephants is illustrated in the story "Elephant Walk," which was made into a movie in the 40s, or perhaps early 50s. This is the true story of a British colonial villa which was built across a traditional elephant walk in India. The elephants were confused and enraged to find their way blocked, and every year at the same time, the villa staff would have to defend the villa and deflect the herds around it so they could get from one side of the offending villa in their traditional migration.
Finally the elephants could not be dissuaded, and one year in rage they finally stormed on through the villa, destroying it and killing many of the people, restoring the traditional road.
In addition to their travel and burial patterns, it has been demonstrated in the 20th century that elephants also have a high friendship skill with humans, as well as develping lasting relationships with other elephants, and in fact do remember individuals of the human and their own species even when separated for decades.
(Google Search) ^^