A 'circle' could mean an unofficial association of people or groups sharing the same interests, activities or achievements such as artistic circles ,family circle, golf circle etc....
So he means they don't mingle or socialize in the same groups or with the same people.