Hello Tako, have you seen Harry Potter? It's the first thing that came to mind when I read this. The scary mystical creatures often whisper words that can not be understood, it sounds scary but you can't hear the words. The action of this is the small effort used to say these words coming out of their mouth, with a whisper. Rags means "The remnants of something", so in this example they look like a small remaining quantity of smoke. So, "Whispering rags of smoke" simply means the sounds of whispering coming from these creatures that look like a small quantity of smoke in the air.

At first I interpreted the sentence like Kathleen does, but on second reading I agree with Tom. “Whispering” her is an adjective that modifies “rags.” It is not a verb with “rags” as its object. Another translation: The misty creatures were rags of smoke that spoke in whispers, and I knew that they were spirits of the dead.

'Rags of smoke' here refers to the appearance of the creatures, i.e. they looked like ragged clouds of smoke. The whispering I think means that they were speaking inaudibly, making a kind of whispering sound the narrator could only just hear.