If your situation was the same as mine, how did you overcome that?
I didn't speak with a native English speaker till i was 21 years old. I definitely felt nervous about my shortcomings in speech initially. Because while i could speak perfect English inside my head, the words wouldn't want to come out. Generally over time, i learned to not feel embarrassed and afraid if i made a mistake, because i realized it's not my native language. I kept stumbling and stumbling before i finally finessed my conversation skills to the point, where i can definitely hold my own in a conversation if needed. So, my general advice would be to keep going and talk more even if you stutter in between, because stuttering is a building block to fluency. You'll learn to not make the same mistake twice.
Edit: Also be aware that the person on the other side of the phone is not a monster, who will gobble you up if you make a mistake in English. People generally understand and are willing to compromise for 2nd Language shortcomings.
Yesterday I asked my co-worker so many times to repeat that lol. Well, but I just have to keep on going.