Ok I've thought a bout it some more. You could use the first sentence, but it makes me think of my dad talking about 'swimming' as an event in a weekly schedule, like every Tuesday evening. Eg. Sorry we don't have time for swimming this week. If it's not as part of a regular schedule you have in your diary, you'd probably say we don't have time to go swimming.

'If we hurry, maybe we'll have time to go swimming' sounds better In this context I think 'time for' is more used for nouns rather than gerunds. E.g. maybe we'll have time for dessert/coffee/a quick chat. The second sentence sound fine.