"They differ in appearance" and "They differ in their appearance" both sound natural to me, although "their appearances" might be more correct. In either case, it is clear that you are saying that the appearance of each individual may differ from the appearance of each of the others. However, when you say "they differ in the appearance", it confuses me. It causes me to wonder: "The appearance of what?" They can differ in a quality like "appearance" but "the appearance" is a specific appearance of one thing, not a quality.

They all sound unnatural. "Non-unique" is not very descriptive, and it's not a commonly used adjective in English. What are you trying to say?

I believe you can't use an article because APPEARANCE used this way is uncountable. You would only use an article when using the definition of APPEARANCE to mean an occassion when something appreared, e.g. The appearance of the teacher in the doorway of the classroom silenced the unruly students.

"All non-unique horse differ in 'the' appearance only" is unnatural because of way there's a "the" in the third sentence.