It's grammatically correct. The it does refer to New Moon - the actual film, not it's name. Because whatever 'it' is 'was released', it must mean the movie as this is the only thing in the sentence that can be released. You could add a few words to make it easier to understand: American ticket sales [for] the first weekend [that] it was released made "New Moon" the third best-selling movie on record. You could also swap the pronoun and the noun: American ticket sales the first weekend "New Moon" was released made it the third best-selling movie on record. The original version sounds better, though, as it keeps the title closer to the important thing we want to say about it.

I assume that the movie had already been mentioned in a previous sentence. If that's true, then "it" is referring to the film. If the film hadn't been mentioned in a previous sentence (which I doubt is the case), then "it" would be a pronoun that didn't have an antecedent- which would be an error.

Yes, "it" refers to the film. Your sentence would be better with a preposition to connect "ticket sales" to "the first weekend". There are several prepositions that would work well: "ticket sales during the first weekend" "ticket sales for the first weekend" "ticket sales on the first weekend" "ticket sales over the first weekend" It is permissible to omit the preposition but the sentence is clearer if you use one.