1. "You should take a rest"
This is how you give someone advice. It means it that you think it is advisable for that person to take a rest.
A doctor would say this to a patient, a parent would say this to a son or daughter (and vice versa), or a friend would say this to another friend. You wouldn't normally say this to your teacher or boss unless you had quite an informal relationship with them.
2. "You better take a rest"
Don't say this - it is bad English. Just look at it - there isn't even a verb in the sentence, is there?
The correct version of this is "You'd better take a rest". The 'd' is a contraction of 'had'. For this to be correct English, you need to include the 'd'.
"You'd better take a rest" is a stronger statement that 'should'. The implication of 'You'd better..' is that there may be bad consequences if you don't take a rest. For example, we often say "I'd better go now", meaning that if I don't go, I'll miss my bus or be late for work.
3. "You could take a rest."
This is just a suggestion. Whereas 'should' is your opinion, 'could' is not your opinion: it is simply a statement. You're saying that it's possible for the person to take a rest if they want to.