I'm not a grammar expert - hopefully some will respond. I think it's understandable, but not quite correct. To my hear, the gerund sounds like it should refer to the subject. But WE were not raining. So, something like this would be correct: Despite sleeping too late, I was able to get to work on time. (SLEEPING refers to I) Despite eating an entire pizza, we were still hungry. (EATING refers to WE). If you're not sure if it's correct, add a phrase to clarify: Despite the fact that it rained last night, we could get to the North.