"It's not near here" is fine.
But - and I realise that it was probably not intentional - the rest of your sentence is quite rude: "I'm going to be late for work because of you".
It's better to say, "I must go now or I'll be late for work" / "Sorry, I can't help you, I'm in a hurry".