It's grammatically correct.
The it does refer to New Moon - the actual film, not it's name.
Because whatever 'it' is 'was released', it must mean the movie as this is the only thing in the sentence that can be released.
You could add a few words to make it easier to understand:
American ticket sales [for] the first weekend [that] it was released made "New Moon" the third best-selling movie on record.
You could also swap the pronoun and the noun:
American ticket sales the first weekend "New Moon" was released made it the third best-selling movie on record.
The original version sounds better, though, as it keeps the title closer to the important thing we want to say about it.