"Irrenhaus" is in colloquial use, and would be used by a German speaker in informal situations. If you want to translate "Madhouse", doesn't that have a similar connotation?
Anyway, neutral terms are "Nervenheilanstalt" or "Psychiatrische Klinik"/"Psychiatrisches Krankhaus" (mental/psychiatric hospital).