You can say - I prefer to have tea or I prefer having tea. It would change the meaning a little - so what exactly do you want to say? I would prefer tea, please. Thank you, you are so kind. (And then you drink your tea!) Or, do you mean, I prefer having tea in the garden in the mornings (rather than in the dining room). This last sentence could also use the 'to have' form.