"So" performs two things at once: (1) it refers back to something previously mentioned and (2) it acts as an adverb.
Determiners and pronouns like "this", "that", "who", "whom", "whose" can do #1, but they cannot do #2, so "so" is sort of a miracle word. Compare these two items:
A) "The hole was small. Mary told me so."
B) "The hole was small. Mary told me that."
"So" and "that" both refer to the entire sentence "The hole was small." The difference is that in (B), "that" acts as a placeholder for "the hole was small". Thus (B) is the same as
"The hole was small. Mary told me the hole was small."
However, in (A) "so" acts as an adverb to modify the verb "told". Thus (A) is similar to
"The hole was small. Mary told me likewise."
In your example, "so" refers back to "little" and modifies the second "was".