Yes, there is a difference. For example, if you call metropolises such as Los Angeles, London, New York or Shanghai towns, you're likely going to get laughed at. It does get a little complicated though because sometimes rather small towns are called cities.
If you are driving and see a sign that says "City Limit," it could be a small town but city is just used here because there really aren't signs that say "Town Limit." (At least I've never seen a "Town Limit" sign). Some major cities get nicknamed as towns (for example Chicago, which is a big city, is nicknamed "Chi-town."