The difference is determined by the definitions of 'compare' and 'contrast'. To compare is to describe the similarities between two things. To contrast is to describe the differences between two things. So 'compare' and 'contrast' are antonyms. So 'in comparison with' describes similarities, and 'in contrast with' describes differences.

With "comparison", you can refer to similarities or differences. But with "contrast", you usually refer to differences. In comparison to the Sun, the Moon is very small. In comparison to a cat, a rabbit is about the same size. In contrast with the Sun, the Moon is very small. In contrast with a cat, a rabbit is about the same size. <--- incorrect because you don't use "in contrast with" when describing similarities.

A comparison tells how different things are the same. Contrasting explains the differences and what is unique.