-아/어/여 버리다 vs. -고 말다
I believe -아/어/여 버리다 and -고 말다 mean basically the same thing except that the first is more common in spoken Korean and has a little more feeling and the second is more common in written Korean.

1) Should I avoid using -고 말다 in spoken Korean?
2) What about other forms like -고 말겠다 and -다(가) 말다 in spoken?
3) Can I use it as a modifier, like: 먹다 만 바나나 in spoken?

감사합니다!

Question

-아/어/여 버리다 vs. -고 말다
I believe -아/어/여 버리다 and -고 말다 mean basically the same thing except that the first is more common in spoken Korean and has a little more feeling and the second is more common in written Korean.

1) Should I avoid using -고 말다 in spoken Korean?
2) What about other forms like -고 말겠다 and -다(가) 말다 in spoken?
3) Can I use it as a modifier, like: 먹다 만 바나나 in spoken?

감사합니다!

Answer

1) Should I avoid using -고 말다 in spoken Korean?

No, There is no reason that we should avoid using it in spoken.

2) What about other forms like -고 말겠다 and -다(가) 말다 in spoken?

The expression "-아/어/여 버리다" is used when emphasizing the completion of the main verb, the completion of action or act of the verb.

Whereas, The expression "-고(야) 말다" is used when emphasizing the result or state of having done. It means "to come to the end with the act or action of main verb , or finally, the act or action of main verb comes true"

In other words, aux verb "버리다" emphasize the act or action, but aux verb "말다" emphasize the state or result/effect.

for example,

As I turned around,  I accidentally hit him in his face.
몸을 돌리다가, 제가 실수로 그의 얼굴을 쳤어요.

그의 얼굴을 쳤어요.
그의 얼굴을 쳐 버렸어요. 
그의 얼굴을 치고 말았어요.
그의 얼굴을 쳐 버리고 말았어요.
그의 얼굴을 치고 말아 버렸어요. (X)

"-고(야) 말겠다/말테다" is used when saying speaker's firm will or decision to   achieve/complete something without fail.

In the expression "-다가 말다", the verb "말다" is not used as aux verb. 말다 means "to stop or quit" in this case, as you know.

3) Can I use it as a modifier, like: 먹다 만 바나나 in spoken?
Yes, why not?

바나나를 먹다(가) 말다, 
먹다(가) 만 바나나.

In this phrase the verb 말다 is not used as aux verb, you know.