No, it means he is taller than you, but not too much taller. The correct sentence is "he is a little taller than me".