They are both contractions of "I have not done it before" so there is no difference whatsoever. One or the other may feel more comfortable to some people. For what it's worth, rarely do I say "I've not".

Modern American English is either ‘I haven’t done it before’ or, equally good, ‘I’ve never done it before’. I think ‘I’ve not done it before’ would be understood but it isn’t natural. Maybe you would say it if you were deliberately trying to sound old-fashioned.

If i don't mistake, but "I've not" and "I haven't not" have the same meaning. "I've" is a short from "I have"