No, don't say that word! :)

No, you are not being impolite. I think it's fine. These days, it's a very mild expletive, and I don't know anybody who would be offended by it. We have a lot of words in English which are far more offensive than 'damn'!

In general, the word "Damn" can be used in a variety of situations. For example, some people upon hearing bad news or something sad, quietly say "damn..." Or, used in anger, "damn it!" ... or even excitement: "Hot damn!" Used in the proper circumstance, it's usually acceptable to say. But be mindful of the people around you, and the environment you are in.