It is an expression that means : "That is the major issue or idea that explains and clarifies a certain situation or action" Example : What is the whole point of this quarrel? ( means ,what is the main reason for this quarrel ("quarrel=fight")

我好像一直以来都认为, point 有 opinion, idea 的意思. 不知道这想法哪里来的

It means "that explains it all".