Like in any language, non-native speakers often struggle with unfamiliar sounds. English doesn't have tones, so saying and hearing them takes a lot of practice. To most English speakers that haven't been studying Chinese for some time, pinyin 'x' (which is similar to English 'sh') and pinyin 'sh' sound the same. In addition, a lot of Chinese speakers can't distinguish 's' from 'sh' and 'x'. A lot of native Chinese speakers mix them up even in Chinese, and just look at how often 's' sounds in English are incorrectly transliterated to 'x' or 'sh' in Chinese. I can't imagine any English speaker would ever say 's' instead of 'sh'/'x' or vice-versa (although you'd likely hear pinyin 'x' instead of 'sh' a lot), but a lot of Chinese people can't pronounce or hear the difference between the two. In addition, English speakers would have trouble with the tones without practice. So basically, the difference between 四 and 是 is obvious to English speakers but not to a lot of Chinese speakers, but the difference between 四 and 死 (or 是 and 十) is obvious to Chinese speakers but not to most English speakers.

Probably because you are not asking the right persons... P!

I'm from South China. I pronounced "shi" as "si" untli I went to college because we seldom speak Mandarin but dialect at home. We just spoke Mandarin when we read Mandarin textbook in school. In some cases, in order to avoid misunderstanding, we will count from one to the number you want to tell after you finish the sentence. So not only some foreigners but also some Chinese cannot distinguish the difference.

It`s only my oppinon but according to my experience, even chinese people (mostly on the south part of the country) can`t pronounce it well. They pronounce both "si" but I guess their tones are correct so they can understand each other. But it could be quite hard to understand for a foreigner.

For me the difference is fairly obvious. As someone mentioned I may have trouble with Shi(1st tone) getting mixed up with xi3. So I have to be careful not to say xi3 fu1 when trying to say shi1 fu1