"Take a break" is more informal, and generally means stop what you're doing, possibly to rest or to do something else for a bit. Some examples:

"Would you like to take a break from your exam studying, and come with me for coffee?"

"When I take a break from this work, I'll make that phone call"

Taking a break usually implies a break from work. 

"Take a rest" usually suggests more explicit rest (physically relaxing the body). And you would usually say "have a rest", so for example: "OK, do five more push-ups, and then have a 30-second rest"

Or if someone is ill: "Maybe you should have a rest" (meaning sit down or lie down). 

This shouldn't be confused with "give it a break" which is a slightly harsh put-down; say someone is arguing a point and they are going on and on, another person in the conversation may say "give it a break", which has a tone of impatience or irritation. 

They both mean the same thing.  You can use either.  You'd be more likely to hear have a rest rather than take a rest, but take a rest will be understood.

Have a rest could mean literally have a rest, as in from playing a game to get your breath back.  Sit down and have a rest.

Take a break could mean to have a rest from something over a long period of time — getting away from work or something for a while.

I hope that helps.