From a conversational standpoint it can be like the difference between coming across as abrasive. But it also depends on how you say it in your tone. This is from the Mexican-spanish perspective. Calmase is more formal and probably the least abrasive way to tell someone to calm down. Tranquilo/a is a nice way to do it and also more formal in that it’s a fancier word in that in can be also used to say slow down or maybe has a bit more use than calmase/cálmate. Cálmate is the least formal so it can come off as more abrasive but also not at all if you dont mean it that way and dont come across that way in tone. A fun note: sometimes you’ll hear a latina mom (my mom said it growing up) or anyone wanting to calm someone down asap in an abrasive way or being mean or as a joke say: “Te calmas o te calmo!?” “Are you going to calm down or should I calm you down?!”

Cálmese,(fml) Cálmate(inf) vs tranquila(o)(colloquial) = remain calm or calm down (depending on tone.) Tranquilo is less rigid and less demanding, not as harsh.