恐ろしい includes a sense of awe. Also, 怖い is subjective, and 恐ろしい is more objective, my dictionary says. Back to your example, you can use whichever, but 怖い怪物 sounds that the speaker himself knows ABOUT the monster very well through his experience or some sort, so colloquially 恐ろしい怪物 is more common.