I would not expect a Japanese work colleague or acquaintance to add "san" to my name while I am here in the United States (it is not a title that I am accustomed to being called, so it would feel odd to me...although in an interesting way, not unpleasant).  And if I traveled to Japan for a work trip, I would not expect the people in my company's Japanese office to all of a sudden address me with the more formal "san" just because I am temporarily in their same location.

However, perhaps I might feel differently if I moved to Japan and was living there and working there for many years.  If I felt like Japan was my new home country, I might wish to be talked to the same way as all the people around me.  For example, what if Bob worked for many years in an office in Japan and everyone spoke to each other formally ("Good morning, Kaori-san") and then they say to him, "Hi Bob" or "Good morning, Bob."  Maybe Bob will feel disappointed that he is treated differently, as if everyone is always focused on the fact that he is a different ethnicity or from a different country.  Maybe that will make Bob feel like he will always be treated like "the outsider", the foreigner.  Or maybe Bob will think that the casual way of addressing him by only his first name causes him to lose face in front of others.  Bob could worry that other people will think he has a lower status in the company when they see that he is not being talked to with the same formal politeness.

I don't actually know.  I am just thinking of a reason that someone might be sensitive about not being called with -san added to their name.  Maybe the person who said it was rude has had some kind of experience like that.  Now he assumes that all people want this formality, and maybe they don't.

If you are unsure, I suppose the best thing would be to ask someone if they prefer that you call them with the +san or by just their first name.

Thank you for the comment, Sonya san.

He didn't refer to the conversation with his colleagues, and he talked about his language exchange friends on other language exchange app.

He feels he didn't be respected by the person who didn't say his name without "san", I guess.

We usually say like this with respect, but on the other hand, it is used for expression to keep distance between the speaker and the listener.

For example, I usually use polite Japanese when my husband and I have an argument. It means we are like strangers, I am not your wife, leave me alone, kind of like this...I hope you understand what I said..