It is not 100% wrong, but it doesn't sound right to a native ear. Abysmal does occasionally mean very deep... but mostly it is used to describe extremely bad; appalling. I would say something like, "The lecture today in math class was abysmal, I couldn't wait until it was over." If you would like to describe Mathematics as a never ending pursuit, one might describe it as: unfathomable to comprehend where you can stop learning in math. Or the pursuit of knowledge in mathematics is unending. Since mathematics isn't a tangible "thing" it is easier to talk about the learning or the amount knowledge available in Mathematics. Though if you did just want to say "Mathematics is a bottomless pit", anyone who studies math will understand your meaning completely while others may ask, "What aspect of mathematics is a bottomless pit?" And you would have to clarify by saying that the you have studied for 5 years and you still feel like a newbie. (Just in case you don't know the definition of newbie, it means someone who is just starting out. New, knows very little.)