Both express surprise and a little bit of shock.
"What in the world did you do last night?" doesn't contain any bad language. It could express simple curiosity. It could express irritation or anger. It would depend on the tone of voice and the context.
"What the hell did you do last night?" is not really OK. ""What the hell did you do last night," is rude, not because "hell" is a bad word, but because the entire sentence expresses anger.
"Hell" is borderline. A hundred years ago it was considered "profanity" and rude. Now, the word itself isn't rude. It depends how it's used.
"You did a hell of a good job!" is informal but fine. I'd love to have my boss say that to me.
On the other hand, "Go to hell!" is angry, and therefore rude. I'd hate to have my boss say that to me.