Very good question! Not sure I have a perfect answer to that one, but let's try.
Aller means "to go", plain and simple. Je vais ici, je vais là-bas, it really means to go somewhere. It can be a country, a shop, your friend's house, etc. It is also used in other expressions like cheering (allez, allez) or in some fixed expressions like "je vais y aller", which usually means "I am leaving".
Se rendre means the same thing but is really just more formal. Je vais en France OR je me rends en France mean the same. Here, I guess "se rendre" sounds more like a real trip than just using "aller".
Like in: I am going to the shop: "je vais au magasin". The following: "je me rends au magasin" would seem pretty weird, so your 2nd point is pretty valid, "se rendre" is more used when doing an important travel, not just going somewhere near.
You are okay if you always use "aller". Side note "se rendre" also means "to surrender" ^^, like "Je me rends, j'abandonne!"