It means that not only were they killed they were brutally massacred with only few survivors left, you would use this when trying to depict a image in a more creative way

Yes. "Whom" is correct here. The writer was being careful with grammar. In speech or in casual writing, many people would use the word "who," and it would not change the meaning. "Whom" makes it clearer. "Whom" is the direct object. It means "troops virtually annihilated the Albigensians." "Nihil" is Latin for "nothing at all." Nihilism is the doctrine that nothing at all is important, or nothing at all exists. "Annihilate" is pronounced "uh-NYE-uh-late" and means destroy completely, literally nothing at all left. "Virtually annihilated" means "maybe more than zero were left, but it was almost annihilation." The sentence is phrased this way because the writer wants to emphasize the most important thing. The important thing is that the Christian crusaders were not just killing Muslims far away, they were also killing people nearby. They were killing Christians in France, people of the same religion who differed on some points of theological doctrine. The writer first states the main point--the church leaders were conducting crusades against nearby Christians in France. Then, in case the reader doesn't know just how bad it was, the writer adds a parenthetic remark. "They crusaded against the Albigensians. By the way, they almost annihilated them." The writer could have written "the missionary zeal of church leaders led to ‘crusades’ nearer to home, and in the name of Christianity virtually annihilated the Albigensians in south-west France, as well as attacking other so-called 'heretics.'" However, the writer's main point isn't "the Albigensians were annihilated." The writer's main point is "they even crusaded against other Christians." The Crusades were a terrible thing. In my high-school history class in the United States they were presented as a terrible thing. However, because they were once considered holy, unfortunately, the word "crusade" is (with a small "c") is commonly used in English to mean "a noble effort for a good cause."

It means they were nearly all killed. Most of them were killed by the Christian troops.

yes it means 'heretics (including Albigensians)' were killed. (If you're asking about 'whom', it shouldn't be used here, in my opinion. 'who' is fine. 'Academics' think "whom" should be used with objects but it's not used like that in real English and sounds strange.)