It doesn't mean anything! B is dismissing what A said about math as nonsense. When B says 'Math Shmath!' he means something like 'Oh, stop talking about math. Math has got nothing to do with this'. He doesn't believe the mathematical reasoning that A has given, and feels that actual evidence is needed. This response is typical of an American Jewish style of humour.

"Math, shmath." basically means, "Math? What nonsense! I don't care about that." We repeat the word with "shm-" or "schm-" as the first consonant to show our disdain. To me, this is an Americanism. I'm not sure, but I think there's a Yiddish connection somewhere. Perhaps one of the US members can throw more light on this. Math, schmath! Science, schmience! Legality, schmegality! Work, schmerk! Boyfriend, schmoyfriend!