In the United States this can be an incredibly complex question.
The short answer is that in ordinary speech a town is smaller than a city. There are no precise definitions but 3,000-10,000 would likely be called a "small town," 10,000-50,000 a "town," 50,000 and up a "city."
Cities are sometimes affectionally called "towns" as a sort of diminutive--"New York, New York, a heck of a town/The Bronx is up and the Battery's down." In every city or town, big or small, the center is called "downtown."
The long answer is that words like "town" and "city" may have specific legal meanings--all of which relate to state law and therefore are different in each of the 50 states!
For example, in most states, "towns" are limited size and there is "unincorporated space" in between. There is land that does not belong to any town.
In the New England states, however, all of the land area is divided up into "towns" or "townships." Every square inch of land belongs to some town, and the instant you leave one town you are in another town.