Yes, it sounds natural. But, fyi, it can mean two different things. If you already returned the books late, it is a sort of apology: “I didn’t want to return the books late, but I was sick so I could not return them on time.” Or, if you returned the books a day early and somebody asks you why, you could say the same sentence to explain why you returned them early (that is, because it was important to you not to be late).