'That they can't do it' is not defining which specific problem they can't do. 'It' is the thing they cannot do and we don't know what 'it' is. That would be known from context or from an earlier sentence. The meaning here is - they can't do *something* . The fact that they can't do this thing is a problem (just one of the many problems that exist).

The article “a” is used because “problem” is a singular, countable noun being introduced for the first time. The clause “that they can’t do it” explains the nature of the problem but does not make it definite or uniquely identifiable. Since the listener is not expected to already know which specific problem is meant, the noun remains indefinite, so “a problem” is correct rather than “the problem.”