I am not 100% sure, but I have heard that it has something to do with the origin of the word. I mean there is a kind of "general" rule that words which come from Greek take some prefixes while those which come from Latin take others.

The common negative prefixes in English include: un-, in-/im-/ir-/il-, non-, de-, dis-, a-, and anti-. Some of these prefixes can function both as negative prefixes and as non-negative prefixes with other meanings. Some root words can take more than one of these prefixes as negative prefixes; the prefixed words have different meanings based on the choice of prefix. And sometimes the choice of prefix is arbitrary, and you just have to learn it individually. If you use a search engine to search for the phrase "negative prefixes in English", you may find many links to sites that have descriptions of the types of meanings that the common negative prefixes can give when added to a root word.

Interesting question. As I know there is no any rules. I just remember these words.