It is ok as a second conditional sentence but 'feel sorry for' suggests pity or sympathy and why would you feel sorry for yourself in this case? Do you mean "... we would both regret it" ? which sounds more logical to me.