I believe that most native English speakers would not be able to clearly state the difference between "speak with or speak to". I believe that "speak to" indicates a single speaker (i.e. one speaker and one listener) and "speak with" indicates two speakers (i.e. speakers and listeners taking turns speaking and listening).
The following is an excerpt from "Practical English Usage", written by Michael Swan and pulished by Oxford University Press (ISBN 0 19 442146 5)
Speak and Talk
1 little difference
There is not very much difference between speak and talk.
In certain situations one or the other is preferred (though they are usually both possible).
Talk is the more usual word to refer to conversational exchanges and informal communication.
When she walked into the room everybody stopped talking.
Could I talk to you about the football match for a few minutes?
Speak is often used for one-way communication and for exchanges in more serious or formal situations.
I'll have to speak to that boy - he's getting very lazy.
They had a terrible row last week, and now they're not speaking to one another.
After she had finished reading the letter, nobody spoke.
3 lectures etc
Talk is often used for the act of giving an informal lecture (a talk); speak is preferred for more formal lectures, sermons etc.
This is Mr Patrick Allen, who's going to talk to us about flower arrangements.
Professor Bowen is going to speak to us on recent developments in his research.
The Pope spoke to the crowd for seventy minutes about world peace.
Speak is the usual word to refer to knowledge and use of languages, and to the physical ability to speak.
She speaks three languages fluently.
We spoke French so that the children wouldn't understand.
His throat operation has left him unable to speak.
5 other cases
One usually asks to speak to somebody on the phone (US also speak with).