It's correct. That would mean that the person began learning music on a guitar, but he plays a different instrument or more instruments now. Perhaps he is famous for his piano playing now. I think it would be more common to say "I started OUT in music playing guitar."

Another suggestion to make the sentence clearer: "I started in music by playing guitar". Your sentence is OK, but it sounds a little too condensed.

If you can play multiple instruments and want to say you learned guitar first, you can say "I learned how to play the guitar before anything else." If you want to say that you are just now learning how to play guitar, you can say "I'm just starting out on guitar." Or, if you want to say that you are capable of playing guitar you could say "I play the guitar."

I started playing the guitar.

Yes, it is.