To have something on your mind means to be thinking about something. If you have women on your mind you are constantly thinking about dating women.
So, to take your mind off of something means to distract you from thinking about it.
"Football takes my mind off of women."
I am no longer thinking of women, because now I am watching football.