Oh, I see.
It must be ~はごめんです/ごめんだ.
Yes, it's similar to 遠慮する.
But your emotion that you want to avoid it sounds more stronger when you use ~はごめんです/ごめんだ.
(例)
A:ねえ、来週のパーティにDさんを招待する?
Hey, are you gonna invite D-san to the party next week?
B:うーん、あの人、お酒を飲むとすごく暴れるでしょう。前のCさんのパーティでも、それで大変だったじゃない?あんなトラブルはごめんだから、あの人は呼ばないつもり。
Well, he behaves pretty violently when drunk, you know. They had a hard time because of his bad behavior in C-san's party last time. I don't want such a trouble for me, so I'm not gonna invite him.
=====
ごめん(なさい)for apology is used only for your own doing.
D: あばれてごめんなさい。テーブルを壊してごめんなさい。けがをさせてごめんなさい。
Sorry for behaving badly. Sorry for breaking the table. Sorry for making you injured.