姨妈 can only be used in family. It refers to the older sister of your mother. Suppose your mother has 4 sisters and she is the third daughter in the family, generally, you should call the first daughter "大姨妈", the second daughter "二姨妈", the fourth daughter "四姨", the fifth daughter "五姨" or "小姨". You can also bring her first name into the appellation, for example, one of my 姨's first name is 小平, I call her "小平姨". It all depends on your preference.
阿姨 can be used in calling any female that is older than you by 10- 20 years, OR, the female that is under 50 years old. In family, you can only call your mother's YOUNGER sister "阿姨".