They are all correct but they mean different things. I study myself. - 'I' am the thing being studied - Like: I study English. I study it myself - 'myself' is an emphasis - Could mean something like: I also study it. or, I know what I'm talking about because I have first-hand knowledge. I study it by myself. - I study it alone - Like: I watched the horror film by myself and I was terrified.

I study by myself.

I studied it on my own.