"in point of fact" is a phrase and can't be split.

When you want to say "this is a fact - do not dispute this" "it is a point of fact" draws attention to your point. You might be trying to show a listener where they have missed a key idea they need to consider when you use it.