It means "towards the area in which you live". It could mean "down your street", or "to your town/city/county". It depends on the context. Example: "I'll tell you the next time I'm down your way and we can have lunch".