"It's not near here" is fine. But - and I realise that it was probably not intentional - the rest of your sentence is quite rude: "I'm going to be late for work because of you". It's better to say, "I must go now or I'll be late for work" / "Sorry, I can't help you, I'm in a hurry".

"It's not near here" - this is okay to say "You are delaying me . . " - this is grammatically okay but nobody would normally talk like this :)

lol that person is just asking for direction, you dont have to make them feel bad. Q: Where is this place? A: I am not very familiar with the area. sorry! A: I am not sure. Sorry A: I dont know. You better ask someone else. A: I am not too sure. Your sentences are not wrong btw.