To me, fluency and accent are two different things.  I agree that it's important to be understood, and if you are actually mispronouncing words badly, we can't possibly accept that as "an accent".

How accurate you need to be depends on what you aim to do. If you use English only to communicate with coworkers, then being understood is all you need. However, if you need to give presentations, write texts for publication and speak publicly, then of course your standards need to be much higher.

In general, if your non-English accent is minimal, we will spend less mental energy trying to re-interpret what you're saying. :)

(Part 2 of 2) One additional note. An accent does make us work harder at understanding, and you should compensate by using simple, direct, predictable English. You should avoid trying to use idioms, colloquialisms, and slang words. We don't expect to hear them in a foreign accent. Furthermore, you probably won't pronounce or use them correctly as they are very tied to subtleties of culture.

(Part 1 of 2) A light-to-medium accent does not interfere with understanding. It can even be considered charmingly exotic. Probably the best proof of this would be the career of Arnold Schwarzenegger. At this point, of course, his Austrian accent is part of his public persona, but it says a lot that he not only was elected governor, but was accepted in screen roles.

It is very rare for people who did not learn a language before, say, adolescence to lose their accent completely. 

Speaking simply as a U.S. resident with no teaching experience, I would suggest that foreign speakers attempt to learn and use the actual phonemes of English, rather than doing what we typically do when we are starting out, which is to substitute the closest available phoneme in our native tongue. The problem is not so much mispronouncing a consonant or vowel, we can adjust automatically if it's close to the right one and we hardly notice it. The problem occurs when a learner cannot hear or reproduce the difference between two different phonemes--or when a word is actually mispronounced so that it actually sounds like a completely different word. So you should try to learn, hear, and reproduce all the phonemes of English. It doesn't matter much if you can't get the "th" sound right, that just sounds "foreign," but it can be difficult if, for example, you can't hear or reproduce the difference between "boat" and "vote."