Look up the etymologies. "Passion" comes from a Latin origin, which actually means to endure or suffer. So you can understand passion as a feeling which pushes you forward, despite all odds. "Enthusiasm" goes way back to Ancient Greek, where it means "having a god inside you". So you would use this to focus on a person's inspiration. Sorry if this is a bit in-depth, but the better writers of language are well aware of where their words come from.

If a perfectly legal and doing-nobody-harm enthusiasm is negated, as a result of cruelty, for instance towards a child, it could turn into a very specific passion, named hate. If an enthusiasm is cultivated enough, and a little bit of luck is also present, a passion could stem from that could be inspirational for a lifetime. Passions are not always with us, sometimes they need to sleep quite often, we should not burn the funeral pyre too soon, just do our best, wait, and see.

In this context, the difference lies in degree. A person could have different levels of "enthusiasm", but "passion" suggests greater interest. If a person has passion in relation to something (or someone), then the person lives it and breathes it.