Both are grammatically correct but can mean different things.
"My brother doesn't work" means the speaker's brother is not employed and therefore in general does not go to work or do any work.
"My brother is not working" has two meanings. It has the above meaning for "my brother doesn't work" but it can ALSO mean that my brother is not currently working (but that he may be employed). That is to say, the speaker's brother may hold a job, but right now, today, he is on vacation, is off work, it's not yet his shift, or for some reason is not working at the moment. This second sentence is more ambiguous, so it's exact meaning depends on context.