If you did change "to me" to "for me" the context would change too, because they haven't the same meaning;
+ I don't know how he did it. It's a mystery to me! (You're 100% sure that it's a mystery to you "NO DOUBT"; also, you have no idea whether it's a mystery to other people or no)
+ I don't know how he did it. For me, It's a mystery. (It's kind of you're giving an opinion; "According to what I saw, it's a mystery")
- - -
- When we offer something (from heart ), we use 'for' (I have a surprise for you / she works for her children)
in general when we use 'for' we include motions in what we do; and to whom we're doing it.
- When we give something (to a specific person), we use 'to' (I gave the cookies to Adam / Where is the phone? I gave it to the kids)