Ok I've thought a bout it some more. You could use the first sentence, but it makes me think of my dad talking about 'swimming' as an event in a weekly schedule, like every Tuesday evening.
Eg. Sorry we don't have time for swimming this week.
If it's not as part of a regular schedule you have in your diary, you'd probably say we don't have time to go swimming.