It means you're not giving yourself full credit where it is deserved.
You are underestimating the true value or worth of yourself.
For example you had an interview for a job, but you didn't get ,although you are very well qualified even for a better job than that one. Your friend knowing this tells you :
"Come on, you can apply for a much better job. Don't sell yourself short."
It means also being modest about someone's achievement.
" He doesn't project himself very well and tends to sell himself short."
Another example is that some people sell themselves short in relationships, in the pursuit of love, they get out of their way ,underestimate themselves and bear being verbally, mentally or physically abused.
"In her last relationship with that guy who already had a girlfriend, she was seriously selling herself short."