There are a lot of answers here already. Because both statements are vague, you can see that they mean a slightly different thing to each person.
That said, let's take a look at the literal meaning of each word.
"Little" means smaller than you are now. After the age of 14 or so, you probably weren't that much smaller than you are now.
"Young" is very relative. If you're 75, then being 40 might sound like being young to you. This depends on how old you are and what your perspective on youth is.
One person may say: "When I was young, right after graduating college, I went on a long trip."
An older person may say: "When I was young and our kids were still in high school, I used to read all of the time."
Hope that helps.