We don't differenciate feminine endings from masculine in the past because there are no such ones.
So, if you'd like to specify who Вы are - women or men, you'll need to use words like женщины/women, леди, дамы/ladies and мужчины/men, парни/boys/guys etc.
For example:
Вы, дамы, вчера ушли слишком рано! (Literary: You, ladies, went home too early yesterday!)
_________________________________
- Вы вчера говорили о футболе? (Did you talk about football yesterday?)
- С кем ты говоришь? (Who are you talking to?)
- С этими мальчиками. (To these boys)