You could also use "complicated." You could probably use any of the synonyms from a dictionary but the most common one would be "complicated." Edit: Please read Jennifer's comment.

Sophisticated mathematicians is the term you could use, but highly-knowledgeable mathematicians should also be applicable. It is harder when you are referring to mathematicians (people) instead of maths (an object). Some of the synonyms for maths would be complicated maths, complex maths, and advanced maths. Generally though, I would stick to your original term, as it is the most commonly used one, and the most straightforward. Cheers :)

In American English, “sophisticated” is not a good word for “mathematician”. Neither is “complicated”. You could say “good” “great” “excellent” “highly-knowledgeable” “phenomenal” “gifted” etc.

Thanks, Jennifer