They are both fine. Putting the preposition at the end is perhaps a bit more natural, although some old-fashioned grammarians used to object to this, apparently based on principles of Latin grammar. (I think no one seriously believes this any more.) Actually, what most people would say (but perhaps not write) in everyday contexts is "Who did you speak to?" but those same grammarians would probably object to that too, and on perhaps somewhat more reasonable grounds.

The two sentences "Whom did you speak to?" and "It's the girl whom you spoke to." both end with a preposition. In formal "snobby" English, the rule was that ending a sentence with a preposition was incorrect. If you structure them correctly, in a formal sense, it will become clear to you. "With whom did you speak?" and "It's the girl to whom you spoke." I know that most people do not speak this way in day-to-day conversations but saying it correctly in your head can help you see which way to structure them. So... in formal academic English writing it is prudent to avoid ending with a preposition. Even native English speakers do not often get this one right and nor do they usually care. Another example is who and whom. Who (subject): "Who is chasing that boy?" Whom (direct object): "Whom is that woman chasing?" Whom (Indirect Object) "To whom is that boy running?"

I would personally say "To whom did you speak?" and "It's the girl whom you spoke to" though both are right and can be used so It's up to you :-)