Both are correct grammatically, but each sentence has a slightly difference meaning/nuance. "'I don't believe what he said" = 我不相信他说的话。 "'I don't believe whatever he said“ = 无论他说的话我都不相信。 The first sentences emphasises disbelief of what "he said". The second sentences stresses disbelief of the person himself (it probably refers to someone who is a routine liar). Hope that's clear :)

Your first one is correct for a specific thing said in the past. The second would be right if someone said something in the past, but you don't know or care exactly what that person said. You don't have a good opinion of that person and so whatever they said, you don't believe them. A more common expression with "whatever" is in the present: "I don't believe whatever you say."