"They differ in appearance" and "They differ in their appearance" both sound natural to me, although "their appearances" might be more correct. In either case, it is clear that you are saying that the appearance of each individual may differ from the appearance of each of the others.
However, when you say "they differ in the appearance", it confuses me. It causes me to wonder: "The appearance of what?" They can differ in a quality like "appearance" but "the appearance" is a specific appearance of one thing, not a quality.