I keep hearing people here say they speak with a "neutral accent." There is no such thing in my opinion. Everyone has some kind of accent no matter where they come from or how they speak. I think they are trying to say that they (think they) speak with the same accent as the educated mainstream in their country.
Anyway, of course it will be easier to understand people who speak the same accent as the one you started learning with. And someone who speaks like a typical British person (for example) will help you to understand more people in Britain (or British movies and television) than someone from India or California who speaks with the accent found in those areas.
I'd recommend probably focussing on learning to communicate with people from one region, whichever one holds most interest for you, and then gradually expanding that circle.
Personally, as someone who has spent most of my life living in Canada and the US, I sometimes find English from other areas (Scotland, New Zealand, Barbados as examples) difficult to understand, but after spending some time, I get used to it and learn to understand these accents. This is harder to do if English isn't your native language.