We often say: The structure/framework of society". 

It is a metaphor or a more flowery way of saying society. It's meant to convey that there are many different types of people, cultures, social rules and norms, etc that make up society. The people, cultures, etc are each a piece of thread and together they form the fabric