Either one is fine.
In the United States, in ordinary use, there is no clear line between "town" and "city." "Town" means small, "city" means large, but you can't define it by population.
We say "town" when, for whatever reason, we are thinking of the community as small, "city" when we are thinking of it as large.
We also say "town" when you are thinking of the community affectionately. Although New York has a population of eight million, someone who loves New York might say "New York is my home town." We say "city" when we are thinking of it in a cold, distant, or neutral... or, possibly, as big and exciting. For example, some living in Madison, Wisconsin, population 250,000, might say "We moved from Sun Prairie to the city so I could be closer to work."
If you are about to tell pleasant stories about a place you like and know well, it might be more natural to say "my town."
(People who study government will tell you that in some US states, there are actual legal differences in the governmental structures of "a town" and "a city." The full official name of New York is "the City of Boston," that of Walpole is "the Town of Walpole." It has something to do with charters and municipal corporations and whether it is governed by a mayor or a town manager or a board of selectmen, or something like that. In everyday life nobody pays attention to that).