Speaking for myself only, I personally don't find the term "lao wai" offensive. To me, it's the same as "外国人' in Japanese which literally means foreigner. Which is exactly what I am, if i were to come to China or Japan. However, the word "gui zi" is on a whole other spectrum and it is a derogatory term.

Some foreigners may be sensitive about the term 老外 simply because they're not accustomed to being in China nor hearing everyone in one place call them the same thing. Some assume it must have a negative meaning. As you said, 老外 does not have a negative connotation. In fact, when in the company of Chinese speakers in America, I'll frequently refer to Chinese as 老中 and Americans as 老美. It works the same way and is respectful.

I don't understand why you have to put labels on people. Why not just call them by whatever nationality they are?

我还以为是老外问的……我收回我的话。

it is not a bad world,it's just spoken English.老 is a kind word in 老外，like 老 in 老哥（brother ）and 老乡（the man who is from the same town as you ）and 老鬼（the dear way that an old wife call her old husband ）.and 外 means foreign,so it's a dear word,and the bad word should be 洋鬼子（洋means foreign and 鬼means ghost 鬼子means foreign enemy especially for Japan enemy）or 洋鬼。we use 老外 a lot,it's a good word.if you like my answer you can try to ask me when you can't get a good answer.