"Driving me mad" or "driving me crazy" generally means really bothering you to the point that you can't concentrate on anything else. This use of "mad" means "crazy," not "angry." The phrase is usually used in a negative sense, to describe something happening that you really don't like. Sometimes it is used in a positive sense. Example of a man saying to a woman: "That dress you are wearing is driving me crazy," meaning the dress is making him crazy with desire. "Fancy" means to like or to be attracted to, especially to like someone in a romantic way rather than just as friends. It can also mean to like a thing. Example: He really fancies that flavor of ice cream. In American English, "fancy" is a bit old-fashioned, but I think that the British still use it a lot. "Fancied him like mad" means "liked him very, very much."