You can just tell him: "You don't need to wait for me." There will be various ways to express it using a variety of vocabulary, just make it simple and straight to the point.

You can say: I will finish work late. You don't need to waiting for me