"Don't exert yourself" means don't use too much energy, or work so hard on something that it makes you tired. It is usually said when it isn't too important to accomplish something. "Don't bother" means don't try at all. It isn't worth it! It is usually said if a task is impossible, or it won't accomplish anything worthwhile.

Hi, "Don't exert yourself" is quite a formal phrase and could be used more in a serious way. For example "when you go to the gym today, don't exert yourself too much" but "Don't bother" is quite a sarcastic phrase and implies that the person is in some way annoying you, for example, you have been asking someone to clean their room...after a week they agree and in reply you say "don't bother, I don't care anyone"