'Butt' and 'ass' are both American terms referring to the same body part. Both are very informal, but 'ass' is the more vulgar of the two words. You wouldn't use either word in polite situations.
'Buttocks' is neutral, and could be used in any situation, informal or formal. Strictly speaking, it refers just to the two cheeks that you sit on, not to the ...um... functional part of that area of your anatomy. We rarely use the word 'buttocks', though, because we rarely need to refer to the cheeks specifically. A physiotherapist might use it, for example, when explaining exercises to their patient. It certainly isn't a common everyday word.
NB A more useful term is 'bottom', which is not considered vulgar. You could use the word 'bottom' when talking to a child or a stranger without causing offence.
'Hip' refers to a different body part entirely. Your hip is the SIDE of your body between your waist and your legs. Someone armed with a gun would wear it on their hip, or a mother might carry a baby on her hip. Your leg bones are connected to your body via your hip bones.