A 'circle' could mean an unofficial association of people or groups sharing the same interests, activities or achievements such as artistic circles ,family circle, golf circle etc.... So he means they don't mingle or socialize in the same groups or with the same people.

Going nowhere.

انا عمرى ما قابلت الرجل صاحب القلعة شخصيا احنا مش بنلف وندور فى نفس الدايرة

HI... It means they don't have the same friends...