Yes, it can be confusing. Here are some sentences for context:
Human - I made a mistake, I am only human. (describing 'self' essence as character - male or female)
Human being - Don't talk to me like that, I am a human being! (state of humanness, being a member of the human race - male or female)
Person - You are the type of person I would like to be friends with. (single individual, either male or female)
Man - You are the type of man I could see myself in a relationship with. (adult male)
Man - It is man who has polluted the planet (as in people as a whole - male and female)
Mankind - One small step for man, one giant leap for mankind. (every person, as a collective)
Does this help?