As a same Japanese, I'm sorry to hear this. However, I guess they didn't intend to hurt your feeling nor think you had worked hard. Probably as you are very familiar with the habit, like saying お先に失礼します, and also spoke it fluently, they may have wanted to return words in English. But they could not.
You might have known, お先に失礼します just means you are leaving earlier than people you are giving these words, お先に-失礼します, so you can use this to anyone including your boss. It's fine to use it to your coleagues, too. If you give these words to people who are responsible to you, they might feel it's a little weird. In that case, you can use just "お先に。"