Living "in the mountains" is non-specific. You live in a mountain range. Living "on a mountain" is specific. You live on a specific mountain.
For example, "I live in the Appalachian Mountains (a mountain range), on Roan Mountain (a specific mountain)."