Like in any language, non-native speakers often struggle with unfamiliar sounds.
English doesn't have tones, so saying and hearing them takes a lot of practice.
To most English speakers that haven't been studying Chinese for some time, pinyin 'x' (which is similar to English 'sh') and pinyin 'sh' sound the same. In addition, a lot of Chinese speakers can't distinguish 's' from 'sh' and 'x'. A lot of native Chinese speakers mix them up even in Chinese, and just look at how often 's' sounds in English are incorrectly transliterated to 'x' or 'sh' in Chinese.
I can't imagine any English speaker would ever say 's' instead of 'sh'/'x' or vice-versa (although you'd likely hear pinyin 'x' instead of 'sh' a lot), but a lot of Chinese people can't pronounce or hear the difference between the two. In addition, English speakers would have trouble with the tones without practice.
So basically, the difference between 四 and 是 is obvious to English speakers but not to a lot of Chinese speakers, but the difference between 四 and 死 (or 是 and 十) is obvious to Chinese speakers but not to most English speakers.