In the context of the song, it means "we don't come from a rich family" (and therefore money is not the most important thing in our lives).