Actually both could be used. "Yes.I tried everything.but it didn't work out." This would be used when the speaker knows it was good to end the relationship. "It didn't work" could imply frustration and helplessness of the speaker. In this case, the speaker wanted to stay with joey, but he refused. "didn't work" refers to something specific going wrong, or something physical not functioning properly. The phrase "didn't work out" usually implies unimportance. I hope this helps.

I would use "didn't work out" to describe some the relationship that ended or something that I planned that didn't happen as expected - not in a positive way. If something "didn't work", it means I was trying for success at something with a strategy or tool and was unsuccessful. "He tried to convince the banker to give him a loan by appealing to his sympathy, but it didn't work." "I tried this formula to solve the problem, but it just didn't work." "Honey, can't you give me one more chance?" "No! That line doesn't work any more."