It depends on the context of the situation or conversation.
Looking at a photograph, someone points to your father in the picture you should say, "Yes, he's my father" or "Yes, that's my father" but not "It's my father"
If a guest is at your house and you are sitting in the living room, and someone walks in and you see that it is your father, and the guest says "Who is here?" or "Who was that?" then you would say, "It's my father."