I don't speak Swahili, but I know what you mean from my experience with other Bantu languages. I found it useful to compare other closely related languages, as well as the reconstructed proto-language.
For example, with respect to proto-Nguni, Xhosa has lost consonants from some of its noun prefixes, so that some nouns of different classes seem to have the same prefix, although their adjectives take different prefixes. Comparing Xhosa words with their equivalents in SiSwati, which has preserved these consonants, the difference becomes clearer.
Also, try to study the characteristics of each class in depth. For example, "duka", if I'm not mistaken, belongs to Class 5, which has the prefix "ji-" on words of one syllable, becoming zero on words of more than one syllable.
On the whole, though, I think the most effective way to learn the prefixes is simply to learn each noun together with an adjective and verb, with their respective prefixes. With enough practice they should hopefully start to stick in your mind. The problem is essentially the same as that of learning genders and agreement in the Romance languages (though admittedly a bit more complex).