The phrase is "let go of me". I think the imperative "let go" has more punch of you don't put the indirect object in between. But I suppose the reality is that a person will choose "let go of me" simply out of habit. Can I ask - how do you know it's "preferred"? Do you see it more often? I'm just curious about your comment.