"It develops games to study math" - "to" is part of the infinitive form.
"It develops games to studying math" - to me, it doesn't sound correct. We can use gerund after preposition "to", when "to" is:
a) part of a phrasal verb or some phrasal combination (e.g. I look forward to meeting you),
b) part of a combination with an adjective (e.g. be addicted to doing sth)
c) part of a combination with a noun (e.g. addiction to gambling).
In your sentence I would use preposition "for" - It develops games for studying math".