Don't worry about it! Most non-native speakers will at least have some accent, to varying degrees.
I honestly think that accent is overrated.
Instead, try to work on speaking 'fluently', i.e. expressing yourself in a smooth manner. It is by far the more important thing. I know many people who speak with strong accents, but they can easily be understood due their good, natural flow.
Hope this helps!