The best answer is D because "encourage somebody in something" is standard usage. In your sentence the preposition "in" indicates limitation, i.e inclusion in some qualification or circumstances. Encourage him in his efforts = Encourage him within the scope or range of his efforts. This does not mean encourage him generally, rather encourage him in a defined range of circumstances--in his efforts.
B is also possible, but has a more general even vague meaning. Encourage him with his efforts: a; encourage him generally using his efforts as an example or stimulus? or b; encourage him so far as concerns his efforts? (approximately equal to D).