Exam of maths is more correct, if you tell about the name of the exam, but if you wanted to tell about process in maths, so you may tell that - I am bad in maths. Other second version of Maths exam is correct also. :)

As far as I understand "maths" is used in British English and "math" in American English.

It needs to be "Math exam" or "an exam in Math".

Maths exam that both versions are correct but there is tell that you have to take one

Both variants are correct.