It is impossible to say whether something is "too informal" without knowing the audience. ("Informal" is one word.) But I will say this: the sentence you wrote says nothing. As a whole, the sentence sounds too meaningless for anybody who is trying to communicate with anyone else. That said: "one hell of a challenge" is correct usage. It's emphatic. It's something a person might say to a friend or a peer.

If it's for a class, I would say it's too informal. "Hell" is technically a curse word and theres not many cases where it's okay to curse in academic settings. I would change "hell" into "heck" so it's not a curse.

In a classroom setting, no, I would not say "one hell of a challenge," even in an "informal" speech--unless I had really worked things up to an emotional peak, and could half-pretend I had slightly lost control. To me, it doesn't fit with the mild level of emotion in the rest of the sentence. Even an "informal" presentation to a group is always more formal than joking with close friends. The safe thing is to substitute "heck," although that sounds almost artificially mild. The best solution is to find some completely different wording to express your strength of feeling. Sometimes less is more. You can use tone of voice. "I hope I'm hiding it, but standing here talking about it is a BIG... CHALLENGE to me." Say it aloud and stand in front of a mirror and see if you can get your tone of voice and your body language express the idea. If you decided to go with either "heck" or "hell," you should join the sounds of "heck of a" or "hell of a" so that it sounds like a single word: "heckuva" or "helluva." There are, technically, three kinds of "bad language:" profanity, vulgarity, and obscenity. Obscenity means sex. Vulgarity means bathroom stuff. Profanity, and "cussing," literally have a religious context. A hundred years ago many people in the US took "hell" and "damnation" quite literally. They were references to something so terrible that you didn't even want to think about them. Nowadays, they are pretty mild--they are not on the list of the "seven words you can't use on television." But they are just lightly beyond "informal," they are "bad language."

You could say that it is extremely challenging, or go into a little more detail about why the presentation is challenging for you. If you're gonna say something as bold as the it's a hell of a challenge you have to make sure that you're absolutely sure that your audience will be receptive to it and you have to back that up by either being playful and relate able or acting bolder through the rest of your presentation. I honestly wouldn't but if you really like the idea and believe that it could work then maybe...