I've never heard of a word for it, either. Something like "the letters got messed up" might be the best thing you could say. There's a long English Wikipedia article on "mojibake," so if we had to refer to it by name I guess we'd use the Japanese word, but it's probably not very well-known. I like Rayne's suggestion of "scrambled," but you might have to specify what you mean.