Rubber bands - made out of actual rubber, like an eraser - wouldn't be very good for your hair. You'd be more likely to use an elastic band, wouldn't you? So you could say elastic band, or just an 'elastic'. Or a ponytail band. Or a 'scrunchie' which is an elastic band with a frilly fabric covering.

You can also use "hair tie". I agree that a rubber band would get tangled in your hair.