It would be fine to say "it" in this case, but the speaker probably said "this" because there is (possibly) more than one hat. The speaker is making a contrast between "this" hat (Amy's hat) and some other hat (the one that belongs to the person who asked if this was their hat). "THIS is Amy's hat. The OTHER hat is your hat."

You have a good point. The whole purpose of pronouns it to avoid repetition, so it would sound much more natural to use "it" in the reply. If you really want to use a demonstrative determiner in the reply, using "that" would sound more natural than repeating "this": Is this my hat? No, that one is Amy's. Yours is over there.

Both are correct. Both can be used. There is a very, very slight implication difference, but so slight that there is hardly any difference at all. "This is Amy's" has the slight emphasis on what Amy posses. Amy owns things and the hat is something Amy owns. The focus is slightly on Amy and her possessions. "It is Amy's" has the slight focus on the hat, the "it" that belongs to Amy.

There is no "right" answer. The speaker can use "this", "that", or "it". The speaker decides which word to use based on how he thinks about the hat. If he thinks of the hat as being somehow "close" to himself, he would say "this hat". If he thinks of the hat as further away (such as on the other side of the room), he might say "that". "It" is more neutral and can always be used.