It is because talking about sex is socially ambiguous.
It is hard to talk about sex with someone without thinking about them in a sexual way. It can easily be the opening move in an attempt at seduction (or even rape). You may just be curious about the topic, but the other person does not know.
If someone you didn't know well suddenly asked you "How much money do you have in your wallet?" what would you think? It might be someone who was just curious to know how much money people carry with them. But it might be someone who was trying to decide whether to rob you. If you were not sure, you would hesitate to answer.