Who are you speaking of? = Of whom are you speaking? You can write such question either way. They mean the same. "Who are you speaking to?" = "To whom are you speaking?"

English grammar enthusiasts don't want to end a sentence with a preposition (words such as of, to, or by), so "Who are you speaking of?" is properly written as "Of whom are you speaking?" "Who" becomes "whom" because it's the object of the preposition "of". You are also correct in saying "To whom are you speaking?"