I only know about this from reading Mark Twain's "Tom Sawyer," describing the behavior of teenaged boys in the mid-1800s. Two boys have taken a dislike to each other. They are "spoiling for a fight." They are daring each other to fight:
“That’s a lie.”
“Your saying so don’t make it so.”
Tom drew a line in the dust with his big toe, and said:
“I dare you to step over that, and I’ll lick you till you can’t stand up. Anybody that’ll take a dare will steal sheep.”
The new boy stepped over promptly, and said:
“Now you said you’d do it, now let’s see you do it.”
Drawing a line and daring somebody to step over it is a way of intentionally starting a fight.
So, figuratively, "drawing a line in the sand" (or dust or dirt) means to set a boundary, with two implications:
a) you will fight anyone who steps over it, and
b) you are testing, daring, or challenging them to do it.