A point-to-point is a horse race, originally in the countryside not on a racecourse and originally in a straight line between two points. Now it is a countryside horse race on a course, with fences to jump. Horses wishing to compete in high class jump races like the English Grand National must first qualify in a local point-to-point.
The expression is occasionally used outside of horse racing to indicate a direct movement between two points or even subjects in a conversation.