Even in the case when we are talking about pointing with your finger, there is a difference. Suppose you are standing next to me and I notice a man at some distance pointing TO or AT me. It would not be natural for me to turn to you and say "Look at that man over there pointing TO me" although it would not technically be wrong. The natural word to choose would be "AT". The reason it sounds wrong to say "TO" is that when you are speaking, your position is your current frame of reference. For the same reason, if someone threw a rock in your direction trying to hit you, it would sound strange to say "That person just threw a rock to me".

Hi Sharon! Good question... 'Point at' and 'point to' are interchangeable when we are talking about the physical motion of pointing with your finger to indicate a person or thing. Where they differ is that 'point to' can also have a figurative meaning of 'lead people to believe', 'indicate a possible explanation or cause' (eg. All the evidence points to the same conclusion; High electricity bills point to an inefficient heating system; Her odd symptoms point to an undiagnosed condition, etc.), while 'point at' is generally only used in the literal meaning of 'indicate with finger'.

I think they're used fairly interchangeably. "Please point to what you want" "Please point at what you want" I tend to prefer "point to," but I don't think you would sound funny if you used either. Once you're this deep into things, regional variations, preferences, etc probably dominate. I would just try and copy whatever the native speakers use around you-- if you already are in the habit of using "at" might as well keep with it so you don't have to doublethink yourself when speaking.