"Leave the call" isn't very formal. It's pretty standard English. I haven't heard anyone use "drop off" - but I guess you can. I would think that "drop out of" sounds more natural to me though. I have to admit that I rarely make conference calls, so it isn't an area I am very familiar with. If you wanted something even more informal, you could probably use "slip out of", to make it sound a little more furtive.