1) Should I avoid using -고 말다 in spoken Korean?
No, There is no reason that we should avoid using it in spoken.
2) What about other forms like -고 말겠다 and -다(가) 말다 in spoken?
The expression "-아/어/여 버리다" is used when emphasizing the completion of the main verb, the completion of action or act of the verb.
Whereas, The expression "-고(야) 말다" is used when emphasizing the result or state of having done. It means "to come to the end with the act or action of main verb , or finally, the act or action of main verb comes true"
In other words, aux verb "버리다" emphasize the act or action, but aux verb "말다" emphasize the state or result/effect.
for example,
As I turned around, I accidentally hit him in his face.
몸을 돌리다가, 제가 실수로 그의 얼굴을 쳤어요.
그의 얼굴을 쳤어요.
그의 얼굴을 쳐 버렸어요.
그의 얼굴을 치고 말았어요.
그의 얼굴을 쳐 버리고 말았어요.
그의 얼굴을 치고 말아 버렸어요. (X)
"-고(야) 말겠다/말테다" is used when saying speaker's firm will or decision to achieve/complete something without fail.
In the expression "-다가 말다", the verb "말다" is not used as aux verb. 말다 means "to stop or quit" in this case, as you know.
3) Can I use it as a modifier, like: 먹다 만 바나나 in spoken?
Yes, why not?
바나나를 먹다(가) 말다,
먹다(가) 만 바나나.
In this phrase the verb 말다 is not used as aux verb, you know.