It's kind of strange, because to regret something, it must have already happened. So anything you regret is certainly in the past. When it comes to these two structures, the difference lies how far in the past a given thing may have been. When a statement is very "fresh" or current, perhaps in the last couple of hours or days, you would more likely use "I regret saying that." When the statement is something you said in the past (say, weeks or months or longer), you'd probably instead use "I regret having said that." I'll offer a final thought... The first construct is basically saying "I regret that I said that." The second construct is saying a little more: "I regret that a previous version of me said that." In other words, in the intervening time, the person is suggesting that they have changed (for the better). It kind of distances that person from responsibility for their words, which is reasonable, since conditions likely have changed.

They are identical in meaning and usage