They both show surprise at something bad. "Oh dear" is more often used when the bad thing is not very serious. But it depends on context and personal preference. Some people overuse these kinds of phrases and so they can sometimes have little meaning.

Oh dear is closer to ух ты... whereas Oh my God is more dramatic and equivalent to Боже мой! Also, Oh dear is something you don't hear much from this generation. I would expect to hear an older woman say it, than a young or middle-aged man, unless they were being humorous.

The first is much, much milder and a much better choice if you don't know the other person well, but an American might be more likely to say, "Oh, no" or "Oh my goodness."