In this context, diametrically opposed is being used to indicate the extreme differences in preferences for buildings between common people and architects. This most likely refers to how architects have a specific and refined taste in architecture while common people do not usually bear knowledge in/appreciate such specific details.