Both sentences have mistakes.
1) For the first sentence, it's correct to say "so hard that [result]," but you can't use "that" twice. Your sentence has two different results ("it was so hard that [result] that [result]"). This doesn't make sense.
You can connect the two results with "and" instead. "My math homework was so hard that my mother, who was a math teacher, almost gave up, and she said I didn't need to do it."
2) For the second sentence, it's not correct to say "too hard that [result]." This structure works with "so hard," but not with "too hard." You can fix it by using the word "so" instead: "My math homework was so hard to get through even for my mom, who was a teacher, that she told me to put it away."
Another way to fix sentence #2 is to replace the word "that" with a conjunction like "so" or "and." "My math homework was too hard to get through even for my mom, who was a teacher, so she told me to put it away."