the difference is that 'what relief' is referring to relief in general whereas 'what a relief' is referring to the relief felt about something specific. when you say 'what a relief' it is a short exclamation of the statement 'what a relief THIS IS'.

They are used differently: What a relief! (used as an exclamation) She was surprised at what relief she felt from taking off her shoes. ( explaining, used with "from")

As the others have said, "What a relief!" is a standalone phrase meaning something like "Thank God!" I have never heard "What relief!"used as a standalone phrase although it could exist as part of a larger sentence (as the others have mentioned). If you saw it in a book it was either a typo, or it was not a complete sentence by itself.