The hyphens (-) indicate that the words in question (brother in law) should be treated as a single word i.e. brother-in-law. A lot of people will just write brother in law without the hyphens and not many people would correct them but brother-in-law is the correct way of writing it in order to make it clear that you are dealing with a single noun (brother-in-law) rather than dealing with a noun 'brother' which is then being altered by the words 'in law'. Does that make sense?

The - is a hyphen. In general, it shows a connection between two or more separate words (often adjectives, though not always) and gives them a specific meaning together. If you took one of the words away, it would change the meaning of the sentence or cause it to be meaningless. Sometimes the hyphen is optional, but with the phrase "brother-in-law," it is not. With this type of hyphenated phrase, there's nothing you can do but memorize where to use it.

¿Por qué se escribe "señor" y "doctor" en minúscula, pero se escribe "Sr." y "Dr." en mayúsculo? Sin duda, hay una categoría en el aprendizaje de idiomas que es nada más sino: "that's just the way it is" :o)