The main difference between the two is in terms of the intensity. A bad dream is a dream about something unpleasant but may be forgotten later; whereas nightmare is a dream about something terrifying/disturbing that leaves you emotionally negative.
So something worse than a bad dream would be a nightmare.