I would say: "I'm not responsible for the size you choose".
Also "I don't have the responsibility for the size you choose."
I don't think either of these would be good in retail or if you were trying to be polite .
If the scenario (or example) is in retail, talking to a customer, it is more polite to say something like: "You can choose the size." or "I would recommend _____, but I think it is best if you choose the size that fits you best." or if you are not allowed to make a recommendation or you do not want the responsibility, then: "We usually let the customer decide what size suits them best."