"Brother-in-law" is the proper spelling. As for what you'd call her, most people would just say "sister-in-law". It's not technically 100% accurate, but it's become so common to say something like that that people will think nothing of it. Alternatively, if you want to be really specific, you'd just say "My brother-in-law's sister".

There is no one word to describe the particular relationship you describe. The most direct way is: "My brother-in-law's younger sister."