"Man" means all humankind, of both sexes. "The brotherhood of men" would be understood to include both sexes as well.
The Barbershop Harmony Society, which organizes and promotes a certain kind of all-male vocal harmony, is a fraternal organization and could be called a brotherhood of men. (Only men can be members; two other organizations, the Sweet Adelines and Harmony Inc. promote a similar style of female quartetting).
The Freemasons are a brotherhood of men. Only men are admitted to the society. (There are parallel organizations for women, such as Eastern Star).
(Traditionally, in English, groups of mixed gender are referred to by using masculine terms. In recent years, feminists have raised objections and gender-neutral substitutes are being used more and more frequently).