"He doubted their motives" means "He thinks they are lying about their motives. They say that their motives are A, when their motives are really B." For example, a farmer says to a sheep "I am fattening you up because I love you," and the sheep says "I doubt your motives, I think you are fattening me up to be slaughtered for meat."
In your second example, I'm not sure I understand the the differences between 1, 3, and 4, but 1 is definitely correct. It means they do not believe the accepted doctrines are true. For example, it was accepted doctrine that the earth was the center of the universe and the sun, stars, and planets revolve around it. Copernicus doubted that accepted doctrine.