Imagine that you are having a clear, coherent talk that's well-focussed on a subject. It's as if you are traveling down a road, or a "way." While you are talking about the main subject, your companion thinks of something she wants to mention. It's not quite on the main topic, but it's slightly related to the topic. Figuratively, it's a little bit off to the side of the main topic. It's at the side of the road. It's "by the way."
It is used when the main thread of the talk reminds somebody of something they think they need to mention.
"So our trip itinerary is that we are going to start in Boston, spend a the weekend in New York, then two days in Philadelphia--oh, by the way, your aunt called and expects us to visit her for dinner while we're in Philadelphia--and then on to Washington, D.C."
However, people often say "by the way" when the side-topic ISN'T really related to the main topic.