"Assumption" is the noun related to the verb "assume." When you assume something, or "make an assumption," you are getting an idea without testing it to see if it's true. For example, if I told you that my child wants to become an engineer, you might assume that my child is male. A "hidden assumption" usually means an assumption that you are not even aware that you are making. It's a little bit difficult to give you an example. Hidden assumptions are beliefs we have about the world which we are not even aware of having, and which rise from our opinion rather than from fact. I hope this helps you.

In the context of your question. This means that the mathematician has built some model but without realizing has made some assumptions when he built it that could not be true. For example, a mathematician in Christopher Columbus's time might have built a model travelling long distances but at that time, these mathematicians "assumed" that the world was flat. The made this assumption without even realizing that it was incorrect. So to open yourself to criticism, you also give others the opportunity to correct any "hidden assumptions" that you make in your model that could make the whole mathematical equation incorrect.