If you say "no es mal" the word 'mal' somehow acts as a noun, not as an adjetive.
"No es mal" → "It is not evil" ('evilness'?)
But you are not talking about the quality here, you are talking about moral/ethical stuff.
Anyway saying "no es mal" sound completely weird although is not strictly incorrect in my opinion.
We say "no es malo" when we talk about quality, as you almost said. It works for music, people doing something (musicians, sportsmen, etc), food, entertaining stuff, etc.
We say "no está malo" in a colloquial/informal context to talk about a person who is not sick or doesn't have any illness currently (at least in Mexico).
Under this context "malo" is equivalent to "enfermo".
"El niño no está malo" → "The boy isn't sick"
"No fuí a la escuela porque estaba malo" → "I didn't go to the school because I was sick"
And this is the only case we use "no está malo" in spanish (no other use pops in my mind right now) but remember that is informal.
Note: Be careful with the accents. The words "esta", "está" and "ésta" are not the same. Even if almost always the context will erase any possibility of confusion, there would be some special cases when putting or not putting the proper accent will change the whole meaning.