Grammatically?
I suppose "Whom are you talking to?" and "To whom are you speaking?" are correct. However, they are unnatural. We rarely use "whom" and when we do it's in a more formal situation.
We usually just say "Who are you talking to?" Or "Who is it?"