It's tricky and it depends on both context and convention, as usual. "They enjoyed themselves at the party" is the most common form you are likely to encounter. For example: "They enjoyed themselves at the party and then went to see a movie." "They enjoyed themselves at the party." is a complete sentence in itself, too. Under some circumstances, you might see the 'during' construct: "They enjoyed themselves during the party until the moment that George fell over" . Even with this example, you could use 'at' instead. If there is some grammar rule to cover this, I don't know it.

The only way "in the party" could make sense is if it were a group of people (eg. going out together, or even a political party). It doesn't make sense when 'party' means 'celebration' as that indicates an event, not something that can surround or enclose you. "At the party" emphasises the location - actually being present there. "During the party" emphasises a time frame - it could imply that their enjoyment was limited only to the time they were at the party, or (as Neal suggests) when something happened to stop the enjoyment.

at or during are both acceptable