We say 20% of the time this happens because here we are using time in the sense of the ongoing passage of time so it cannot be plural. We use times when we are talking about instances/cases which can be both singular or plural so in your example 2 instances of 3 =6 I hope this is clear