Say you're living abroad and you're either studying the local language or you're already fluent in it.
Oftentimes, when you meet someone new, they insist on speaking English with you, despite you being perfectly capable of holding a conversation in the local language.
Does that annoy you, too?
Yes, but those who are not native speakers of English are actually very lucky. It's just a matter of saying "Sorry, cannot speak english, only xxx and the target language. So they have no choice.
I'm in Russia now and I always try to use my russian of course but some people, especially young ones working in restaurants, bars and etc try to answer in english as soon as they realise my wrong cases or accent. Then I politely tell them in russian that I can only speak Portuguese or Russian, Then there's no way that they can use their english :)
But native speakers of English will always have this problem.
That happened to me a lot in China and abroad in general. I think often they are just happy to have an opportunity to practice their english!
I was annoyed at first - I thought they meant my language skills were so bad that they had to switch to english to accommodate me. However most of the time they just want to be nice or take the opportunity to improve their skills too. If you tell them that you want to practice their language, I am sure they will be happy to oblige!