A "dork" is a person (usually a young person) who is socially awkward and doesn't fit in with the popular crowd. "Dorky" is simply a way to make the word into an adjective. His dorky smile made his discomfort even more evident than it would have been otherwise.

A dork is someone who is also nerdy. A dorky smile may imply a strange looking face. Sometimes smart people with abnormal social skills are labeled as dorks.