"Mutual boundary determination of A with B' might sound better like this: Mutual boundary determination between A and B...
Or you could use: 'agreement on the location of the border between A and B...'
It's hard to say without seeing the rest of the sentence. If you post the entire sentence that you want to write with that noun phrase, I'll have a better idea of ways to rephrase it.
I hope this helps. Have a great day!