All three sentences are correct. As someone who has done a lot of scientific writing, though, I have to say that the first and third sound more natural than the second. In the first and second sentences, the mesh is doing the filtering. In the third sentence, a person is doing the filtering. The third sentence really means: "Particles as small as 10 mm are filtered out (by the experimenter) using a mesh." I think that the reason the third sentence sounds more natural than the second is that in scientific writing, the passive voice is often used to avoid saying "I" or "we". In other words, instead of saying, "We use a mesh to filter out particles as small as 10 mm," one writes "Particles as small as 10 mm are filtered out using a mesh." If you start reading the sentence, expecting the doer of the action to be a person, and then find that it ends with "by a mesh," it's a little surprising. (This is not a grammar rule but rather an explanation of why the second sentence sounds slightly unnatural.) To summarize, the original sentence: "A mesh filters out particles as small as 10 mm." is the preferred usage, since it's generally advisable in English to write concisely and to avoid the passive. (You can do what you like when you're being artistic, but this sentence implies a scientific context.) The passive version: "Particles as small as 10 mm are filtered out using a mesh." is also OK and might even be preferable if you want to place the emphasis on the particles that are filtered out rather than the mesh. Incidentally, I don't like "as small as 10 mm", since it's a little ambiguous. (Whether 9-mm particles are filtered out is not completely clear.) It would be better to say something like, "A mesh filters out particles larger than 10 mm in diameter," or "A mesh filters out particles 10 mm in diameter or larger." It's a good idea in scientific writing to make things as clear and unambiguous as possible.

Your sentence is absolutely correct. If I were your teacher, I'd give you full marks for it. In fact, by strict grammar rules (the doer of the action is introduced by "by" in passive voice), your answer would have been the only acceptable one. Frankly, I don't agree with the answer in the book. Using a mesh gives me the impression that there is someone else involved who is using the mesh. Whereas with passive voice the mesh is what should be doing the filtering.