Little Lady is usually a sweet or playful way to refer to a girl or young woman, not about her actual size.

These answers are all correct. Nowadays, this term as well as 'my old lady' is used in a derogatory manner referring to someone's wife. People in their late 50's and 60's use it.

"Little lady" can indeed be a term of endearment, but it can also be somewhat derogatory, or sarcastic. Depends on the context, really. In a postive way, "Little lady" is often used in show-biz, like people on stage. As in "A Big Hand for the Little Lady." (a movie from 1966 with Henry Fonda). Or, at a fair, the guy at the stand might say to a man and his (younger) daughter: "Here's your bear, Sir, and some candy for the litlle lady." "Little lady" can also refer to one's wife. As in: "I brought back some bacon for the little lady." Derogatory/sarcastic: "Where do think you are you going, little lady?" Which is what a father might say to his teenage daughter who want to go out late at night. Americans slightly favor "young lady" in that case. Like I said, much depends on the circumstances, really, whether the person is being sarcastic or not.

P.S. I should perhaps also point out the obvious: that 'little' and 'small' are not interchangeable here. 'Small,' when you're talking about people, is said of their height: "The little girl looked very small for her age." Meaning, she wasn't as tall as most other girls her age. Conversely, as I indicated, you wouldn't use 'big' either, but 'tall.' "You can tell me, I'm a big girl!" Means she considers herself grown-up (mentally).