There is no ambiguity at all. The first person says that it works. Then also suggests trying it a different way. The second person does not believe the second way will work.

Not really sure, i'm a native speaker and i'm also confused. It would make more sense if B spoke first, and then A. meaning B doubt it would work, then A showed it worked.

Because of what A said at the start, it does become somewhat ambiguous. A said "it works!" and maybe B just waited for them to finish before saying "I doubt it would work" but that seems like a weird way to reply, especially without further specifying that it working in the first place is what they are skeptical of. So the most likely scenario is that A said "it works, but let's try it in a different way" and then B says "it works, but I think if we change it, I don't think it will work." So "I doubt it" means "I don't think so" or "I don't have any confidence in that [method/procedure/etc.]"

There is no confusion. B believes it would not work if they try it diffently.