Yes, it means "tall, but not too much."
If you were asked to describe him, you wouldn't say "He's really, really tall." Or "he's really, really handsome." He's pretty average. He's a little tall. Kind of nice looking. He looks like everybody else - nothing sticks out about him.