If anyone has taught you to say 'When I was at London', they are wrong.
With names of places, the correct preposition is 'in.'
In London
In Mexico
In Africa
'In' has a similar meaning to 'inside'. So you say 'The milk is in the fridge' or 'The money is in the bag', and you also say 'in' with the names of towns, cities, countries and continents. This is because these places have defined borders.
When you are in Italy you are within the borders of the Italian state. When you are in Rome, you are within the boundaries of that city.
We use 'at' in set phrases such as 'at home', 'at school', and 'at work'. We can also use 'at' for other locations, particularly if you are more interested in the function of the place than the fact that it is an enclosed space. For example, you could say 'Let's meet at the cafe' or 'He's at the bank at the moment. He's getting some money'.
In a few very rare occasions, it is possible to use 'at' with the name of a city, but what we actually mean is something different. If you're on a long train journey, at some point in the journey you might, for example, phone home and say 'We're at Oxford now' . But you don't mean the city - you mean the station. Likewise a student might say 'I'm at Cambridge'. But they don't mean the city - they mean that they are a student at the university, indicating an institution rather than a physical location. If you mean the city, you would say 'in'. For example, you might say 'Last year I spent a month staying with a family in Cambridge.'
I hope that helps.