You use "by way of something" when you are explaining the purpose of something. So "by way of a greeting" means that what he said ("Comb your hair") was intended to be a greeting. Please keep in mind that if this were a normal person, "by way of a greeting" would refer to something like Hello, or Good morning. This rude greeting is only there because the character in question is a vile person like Vernon Dursley.