"By my own" isn't a correctly phrased sentence. "On my own" strongly implies physical solitude of oneself, whereas "by myself" doesn't feel so binding, and could mean, for example "I am studying by myself" which doesn't mean you won't work with others, just for a moment or so you'll be working alone, it doesn't heavily imply that you'll be binding yourself to solitude. "I am studying on my own" would imply that you'll be doing it entirely independently with no help at all.

By myself = correct On my own = correct *By my own = No I wouldn't say that there is a significant difference between 'by myself' and 'on my own'. They can be used interchangeably in most situations.