The correct answer is A. "I spent my holiday fishing." This sentence structure indicates that fishing was the activity you engaged in during your holiday.

C doesn't really make sense. B is grammatically correct, but no native would ever say it that way. A is definitely the best choice, since that's how a native would say it.

A is best and C is wrong. However, suppose you were unhappy with your holiday fishing because you wanted to do other things instead. In that case, (D) "I wasted my holiday by fishing" is better than (E) "I wasted my holiday fishing". because (D) makes it more clear that fishing is the reason your holiday was bad. The reason (A) works is that "fishing" works as an adjective describing your holiday. The reason (D) works is that "by fishing" works as an adverb modifying "spent". However, (E) works less well because "fishing" can be either an adverb modifying "spent" or an adjective modifying "holiday". Since it is closer in the sentence to "holiday", it is likely to be interpreted in the second way, which is not good if that is not the meaning that you intend.