"I don't think that would be a good idea," is one way to use that in a sentence. If I tried to say it with the other phrase, I'd have to re-arrange the words a bit: "I think that wouldn't be a good idea." So, not a big difference, but perhaps a nuanced difference in tone. Honestly, the "I think that wouldn't," is probably not grammatically correct. The first version sounds more correct, to me. "I don't think that would work," rather than "I think that wouldn't work." The second version doesn't sound quite right.

They are virtually identical. Perhaps, "I don't think . . . " is just a tiny bit "softer," more hesitant, or less assertive -- a little more like a suggestion and a little less like a strong statement of opinion.