Departure would not be used with ed. The sentence could be altered to be correct:
I decided to return home soon after I made my departure in my car.
There is still a problem. Departure is often set aside for flights. So I would further change the sentence:
I decided to return soon after leaving in my car.