In the Oxford Learner’s Dictionary, they give two definitions. The first one is uncountable, and the second one is countable. I might suggest that even the first one would be countable, except that, in reality (= “in fact”), there is only one reality, so the word functions almost like a proper noun. In science fiction, they might use some sort of technology to travel to an “alternate reality”. It’s similar with “world”, “earth”, etc.

I see what you mean. I've had a look at a few dictionaries, and they seem to me to be wrong!