It's a little awkward, but the direct way to say it is "If I keep not understanding this course, my money will be wasted."
As others have noted there are probably better ways to say. And regarding Jeff's answer, which is certainly grammatically correct, I think it's worth pointing out that the meaning is different.
"If I keep not understanding" implies that you have never understood and may never understand.
"If I don't keep understanding" implies that you previously understood the material but might be worried that it will become harder and you will not understand in the future.
If you replace "keep" with "continue" I think it's more clear: "If I continue to not understand" has a different meaning from "If I don't continue to understand."
I hope that's helpful!