Being monstrous is usually more in terms of appearance. It can also be used to describe a trait that is to an extreme, like "monstrously tall," in which case it might not have a negative connotation.
Being scary is a lot less specific, but does also mean to cause fear for some reason or other. If someone says "you're scary," it may not be negative either, as this term is sometimes used jokingly--of course, context matters in this case.
They both have similar meanings, but monstrous is a lot more extreme, and usually is pointing out something specific.