"I was talking to her" is correct, and implies that you are talking to a female. "I was talking to him" is correct, and implies that you are talking to a male. "His" and "her" are object pronouns.
"His" is a possessive pronoun. It indicates ownership.
John has a book.
He has a book.
Whose book is it?
It is his book.
"His" is masculine. The feminine form is "her."
Mary has a book.
She has a book.
Whose book it it?
It is her book.
You cannot say "I was talking to his" because "his" functions like an adjective, not like a noun.