The way this sentence is constructed, the word 'which' refers to 'gradualism' and this is obviously incorrect because gradualism means species evolve gradually. If you want it to refer to the first hypothesis you would need to place the subordinate clause beginning with 'which' in the following position in the sentence: The punctuated equilibrium hypothesis, which holds that species evolve in relatively sudden bursts of brief duration, challenged gradualism.

It means "gradualism"