Continuous Subject-in-the-Loop Integration

• CSLI is a framework ensuring persistent subject involvement in every phase of AI development by mandating a minimum normalized involvement score for each impacted subject.
• It employs iterative knowledge graph curation with expert-in-the-loop feedback, using candidate validation via scoring thresholds to refine AI outputs.
• The framework evaluates system infrastructure along three axes—Coverage, Integration, and Outlook—to enforce community control and promote accountable AI design.

Continuous Subject-in-the-Loop Integration (CSLI) is a guiding principle and overarching framework for ensuring that the subjects affected by AI systems are continuously and systematically integrated into all stages of development, deployment, and maintenance. Originally formalized to address persistent failures in centering marginalized communities within AI infrastructure, CSLI has been refined both as a principle for radical community-centered AI and as a concrete methodology in expert-in-the-loop knowledge graph construction. CSLI mandates a nonzero, persistent level of direct influence from affected subjects throughout the entire AI system lifecycle, seeking to guarantee both substantive participation and ongoing control over technological directions (Roewer-Despres et al., 2020, Rahman et al., 2024).

1. Formal Definition and Mathematical Foundations

CSLI is defined not as a fixed protocol, but as an accountability and integration principle requiring that, for every phase in an AI system’s development (denoted by the ordered set $\Phi = \{\phi_1, \phi_2, \dots, \phi_n\}$), each impacted subject $s \in S_{\text{impacted}}$ must achieve direct integration with a normalized involvement score of at least $\epsilon > 0$. Formally,

$$\forall\, \phi \in \Phi,\quad \forall\, s \in S_{\text{impacted}},\quad \text{Involvement}(s, \phi) \geq \epsilon$$

with $\epsilon$ prescribed by infrastructure designers subject to context.

In iterative knowledge graph (KG) curation, CSLI implements a recurrent process in which candidate facts $C_t$ generated by an AI agent are reviewed and evaluated by expert annotators. This leads to a feedback-driven update function,

$KG_{t+1} = I(KG_t, C_t, U_t)$

where $KG_t = (V_t, E_t, L_t)$ represents the KG state, $C_t$ is the candidate set (with confidence scores), and $U_t$ is expert feedback mapping candidates to $s \in S_{\text{impacted}}$0. The update $s \in S_{\text{impacted}}$1 consists of all candidates with scores above a threshold $s \in S_{\text{impacted}}$2 that were approved, i.e., $s \in S_{\text{impacted}}$3 (Rahman et al., 2024).

2. Criteria for Infrastructure Evaluation

To operationalize CSLI and identify relevant infrastructure gaps, three axes are used:

• Coverage: Quantifies the proportion of actually consulted subjects,

$s \in S_{\text{impacted}}$4

ensuring all affected groups (especially marginalized ones) are systematically included.

• Integration: Measures the degree and proactivity of subject incorporation into the development process. Integration is conceptualized as the average involvement across phases and subjects,

$s \in S_{\text{impacted}}$5

although no fixed formula is enforced.

• Outlook: Represents whether the infrastructure supports subject-led, positive, forward-oriented visions for AI. This is a boolean indicator: $s \in S_{\text{impacted}}$6 if such infrastructure is present and explicit, $s \in S_{\text{impacted}}$7 otherwise.

No pre-specified weighting or canonical aggregation is mandated, though the axes form an implicit multidimensional rubric (Roewer-Despres et al., 2020).

3. Core Infrastructure Requirements

CSLI-driven systems require robust, public-minded infrastructure spanning four domains:

Infrastructure Type	Primary Role	CSLI Axis Addressed
Regulatory	Mandates subject involvement and penalizes non-compliance	Coverage, Integration
Legal	Establishes statutory remedies, model audits, recourse	Integration
Support	Federated councils, civil-society networks, best-practice sharing	Integration, Outlook
Educational	AI literacy programs, accessible curricula, workshops	Outlook

Each infrastructure form is necessary to guarantee ongoing, substantive subject influence—from pre-design through post-deployment recourse. Regulatory and legal mechanisms enforce subject power, while support and educational components enable positive, community-led innovation (Roewer-Despres et al., 2020).

4. Domain-Specific Implementation: Knowledge Graphs

CSLI has been concretely enacted in human-in-the-loop knowledge graph construction via systems like Kyurem. The Kyurem architecture integrates CSLI through notebook-based, widget-driven interfaces, enabling domain experts to continually review, annotate, and approve AI-suggested candidates. The system features:

• Modular JS front-end widgets for graph overviews, node-link visualization, and tabular annotation
• Python API exposing KG ops within Jupyter, directly connected to backend microservices (Cypher/Neo4j)
• Synchronized, multi-modal views (e.g., entity distributions, corpus snippets, relations) allowing coordinated, context-rich decision-making
• In-notebook expert feedback, reducing cognitive overhead from context switching and external tool juggling

Workflow stages include phased gap identification, AI-driven candidate expansion, prioritized candidate scoring, expert review (with explicit decision markers), and automated integration into the KG. Aggregation of multiple expert inputs is supported through majority-signed feedback (Rahman et al., 2024).

5. Evaluation Rubric and Case Analysis

The qualitative evaluation of candidate frameworks under CSLI employs the three axes (Coverage, Integration, Outlook):

• Coverage: Scope validation mechanisms that enumerate impacted groups increase coverage but may fall short when group inclusion is discretionary.
• Integration: Mandated consultation and mutual accountability favorably affect integration, yet reactive designs—such as post hoc audits—fail to achieve “day-one” co-development status.
• Outlook: Proposals focused solely on harm mitigation, lacking means for subjects to propose positive AI visions, are rated poorly on outlook.

The accountability framework of Berscheid & Roewer-Despres (2019) was reflexively evaluated under these criteria, revealing strengths in enforced developer accountability, but also highlighting deficiencies in proactive participation and future-oriented community visioning (Roewer-Despres et al., 2020).

6. Limitations and Open Challenges

CSLI presents several ongoing challenges:

• Quantitative Operationalization: The principle is intentionally qualitative and context-sensitive, not a standalone metric. Operationalizing CSLI without undermining community control remains an open area.
• Uptake by Power Structures: Implementing CSLI’s requirements often faces resistance from established institutions with little incentive to cede technological governance.
• Enabling Positive Visioning: Most interventions still focus on risk containment rather than empowering communities to actively reimagine AI utility to further their own socio-technical agendas.
• Global and Cultural Adaptation: The paradigm and case studies are currently skewed towards North American contexts; robust adaptation to global legal, educational, and infrastructural variance is required for universal relevance (Roewer-Despres et al., 2020).

7. Best Practices and Design Guidelines

Insights from practical CSLI applications recommend:

• Aligning interface components and feedback mechanisms with domain-specific graph or workflow task taxonomies
• Embedding interactive human-AI loops in environments already familiar to experts to minimize onboarding and friction
• Ensuring multi-coordinated, multi-modal data exploration supports scalability and heterogeneous sensemaking
• Formalizing integration and feedback aggregation protocols (such as explicit thresholds $s \in S_{\text{impacted}}$8 and versioning) for reproducibility and tractability

Kyurem demonstrated that context-integrated, widget-based workflows can achieve 30–50% speedups in expert review, and increased decision confidence, underlining the practical benefit of CSLI-driven curation (Rahman et al., 2024).

---

CSLI defines a foundational approach for responsible, subject-centered AI, requiring an unprecedented degree of substantive community participation and continual partnership, enforced by a multifaceted, public-centric infrastructure. Its uptake and further formalization remain active domains in both theoretical and applied AI ethics research.