Listen to people from Glasgow, Scotland. If you can understand them, you can understand anyone!
The different accents DO make it hard don't they?! Even as a native speaker I can sometimes not understand people that come from other countries speaking English.
You'll need friends from different places and to watch movies in English from lots of different countries! Even if you watch movies made in America you can hear different accents.
The thing is that then you will also need to learn the slang and curse words for each place too.
The question is, "What is your GOAL?!" Do you want to be able to travel to a certain country or place? If so, then I would concentrate and learning just that dialect and accent. I would look for friends here from that place.
Good luck to you!