I would say "on" is a slang/lazy way to say it. The moment you say "on my..." it implies "on top of my... something", for example "on my left knee". Practically everyone will know what you mean, but there is a small chance that someone cracks a joke about it, or misinterprets until you complete the sentence and they fully digest it. "To" is the proper way to say it. And "to the left side of me" is a fullest, clearest way to say it - useful in case someone struggles to understand, although it's usually inefficient and unnecessary to use so many words.

“On my left” is definitely more common. “To my left” is used sometimes. It has a subtly different connotation, emphasizing directionality, or the act of positioning oneself and/or others in relation to an overall design.

Can you sit on my left