They have slightly different nuances.
I couldn't say.=I don't know.
Example:
A: How long will it take to get from London to Paris?
B: I couldn't say. I have never made that journey, so I don't know.
I wouldn't be able to say.=I don't know because it has not happened yet. Or, I am unwilling to tell you.
Example:
A: You heard that conversation, so how did it really go?
B: I don't think you need to know, so I wouldn't be able to say.