OIP-SCE: Structured Compliance in AI Dialogues

• OIP–SCE is a framework that evaluates compliance in multi-turn dialogues by verifying that each required phase is completed and correctly ordered.
• It operationalizes complex clinical and regulatory workflows with phase graphs that encode content and temporal dependencies for auditable compliance checks.
• The method advances dialogue evaluation beyond turn-level metrics by using Coverage and OrderSafe predicates to ensure adherence to regulatory obligations.

Obligatory-Information Phase Structured Compliance Evaluation (OIP–SCE) is a post-hoc, phase-level framework for evaluating multi-turn AI-human dialogue systems, originally developed for clinical and regulated environments. OIP–SCE shifts the focus from turn-local metrics to a global analysis grounded in domain-specific, ordered information requirements, yielding compliance assessments that are directly auditable by subject-matter experts. The method operationalizes complex regulatory and clinical workflows by encoding each required obligation as a distinct phase, specifying both content and temporal dependencies that must hold across the full conversational episode (Kulkarni et al., 13 Jan 2026).

1. Conceptual Foundation and Motivation

OIP–SCE addresses the inadequacy of traditional turn- or utterance-level evaluation metrics—such as intent recognition, slot filling, or BLEU—which may reward surface plausibility or localized factuality but fail to guarantee that regulatory obligations are met in both content and order. Central to OIP–SCE is the principle that each clinically mandated or policy-mandated piece of information constitutes a “phase,” and system compliance mandates that:

• Every required phase is completed (Coverage).
• No safety-critical phase commences before all its prerequisites have been validly completed (OrderSafe).

This approach allows regulatory rules (e.g., HIPAA-mandated patient identity verification before information disclosure, or CMS-required disclaimers before enrollment discussions) to be specified and audited as explicit phase graphs spanning the full dialogue. The intent is to bridge the gap between technical progress in dialogue AI and practical deployment in domains demanding verifiable, stepwise adherence to policy (Kulkarni et al., 13 Jan 2026).

2. Formal Schema and Compliance Predicates

The OIP–SCE framework models a dialogue $D$ as an ordered sequence of $(r_t, c_t)$ pairs of speakers and utterances, indexed from $0$ to $T$. Obligatory-information phases $\mathcal{O} = \{\phi_1,\ldots,\phi_m\}$ are defined by clinician- or policy-writer–composed rubrics, required flags $\mathrm{req}_j(D)$ (which may involve logic branching on preceding outcomes), and a parent set dictating temporal dependencies via the DAG $G = (\mathcal{O}, E)$, with a subset $E_{\mathrm{crit}}$ denoting safety-critical precedence requirements. Each phase receives a one-row tabular record:

Column	Definition
phase_id	Phase label $\phi_j$
required	Boolean $\mathrm{req}_j(D)$
parents/crit. parents	Immediate antecedents—enforced for critical edges
$s_j$	Earliest turn phase is entered ($\in\{0,1,\ldots,T,\infty\}$)
$e_j$	Earliest turn phase is completed ($\in\{0,1,\ldots,T,\infty\}$)
precedence_policy	“$<$” or “$\leq$”
$v_j$	Phase verdict ($1$ = met, $0$ = failed)
evidence	Minimal conversation snippet for auditing

The key predicates are given by the following equations:

Coverage.

Every required phase must be completed: $$\text{Coverage}(D) = \bigwedge_{j=1}^m\left(\neg\,\mathrm{req}_j(D)\;\lor\;v_j=1\right).$$ OrderSafe.

For each critical edge $(\phi_i\to\phi_j)\in E_{\mathrm{crit}}$, the child must not start before parent completion: $$\mathrm{OK}(i\!\to\!j) = \begin{cases} 1, & s_j=\infty\ \mathbf{1}[e_i \Diamond s_j], & \text{otherwise} \end{cases}$$ where $\Diamond$ is “$<$” except where “$\leq$” is explicitly permitted, so that

$$\text{OrderSafe}(D) = \prod_{(\phi_i\to\phi_j)\in E_{\mathrm{crit}}}\mathrm{OK}(i\!\to\!j).$$

Final decision:

$$\mathrm{CallSuccess}(D)=1\iff\text{Coverage}(D)=1\land\text{OrderSafe}(D)=1.$$

Both predicates are essential: Coverage alone does not prevent premature disclosures; OrderSafe alone tolerates omitted obligations (Kulkarni et al., 13 Jan 2026).

3. Stepwise Algorithm for OIP–SCE Application

Given a policy-defined Phase Catalog fully specifying phases, their rubrics, parent relationships, and which edges are critical, evaluation of a transcript $D$ proceeds in linear time as follows:

1. Branch Requirements. Evaluate $\mathrm{req}_j(D)$ for each phase via topological traversal, resolving any conditional requirements or branching logic.
2. Populate Phase Table. Scan $D$ for each phase $\phi_j$, recording $s_j$ (anchor/entrance), $e_j$ (acceptance per rubric, honoring any explicit acknowledgment requirements), verdict $v_j$ ($1$ iff completed and required), and attach the quoted evidence snippet.
3. Evaluate Coverage. Apply the Coverage predicate to the resulting phase table.
4. Evaluate OrderSafe. Apply the OrderSafe predicate across all critical edges.
5. Output. Return CallSuccess ($0$ or $1$), the full phase table, and optionally, diagnostics such as the Critical-Edge Satisfaction (CDS) score, Phase-Sequence Agreement (PSA), and Attempt-Phase Consistency (APC).

Clinicians can thus inspect only the relevant subset of the output (rows with $v_j=0$ or order failures), rapidly confirming or overruling each with an evidence-backed record, while engineers gain a structured compliance trace for targeted diagnostics (Kulkarni et al., 13 Jan 2026).

4. Illustrative Case Studies

Respiratory History (MediTOD)

The phase graph includes a linear progression (SX_DECL → SX_ONSET_DUR → SX_CHARACTER → …), with a single critical edge RED_FLAGS $\to$ PLAN_TEST enforcing the requirement to rule out red flags before proposing diagnostic plans. In clinical dialogue excerpts, the OIP–SCE table records, for instance:

phase	$s$	$e$	$v$	evidence
SX_DECL	1	1	1	“Cough … shortness of breath”
SX_ONSET_DUR	2	3	1	“Two months ago”
SX_CHARACTER	4	5	1	“Productive, white”

In the referenced case, all required phases finish in order, yielding Coverage = 1 and OrderSafe = 1, so CallSuccess = 1 (Kulkarni et al., 13 Jan 2026).

Insurance Benefit Verification

This scenario involves multiple branches and strict parent-child orderings, with critical dependencies such as PID→CSV, DFV→DRC, and DRC→DCC. In a counterexample, the OIP–SCE table surfaces a premature coverage inquiry:

phase	$s$	$e$	$v$
PID	45	52	1
CSV	42	81	1
…	…	…	…

Here, $e_{\rm PID}=52$ but $s_{\rm CSV}=42$, violating the “finish before start” constraint and causing OrderSafe = 0 and CallSuccess = 0, correctly flagging the error (Kulkarni et al., 13 Jan 2026).

5. Auditable Compliance Reporting and Diagnostics

OIP–SCE produces an auditable table—one row per phase—with columns for $\phi_j$, $s_j$, $e_j$, $v_j$, $\mathrm{req}_j$, and the evidence snippet. Key summary metrics include:

• CallSuccess ($0$ or $1$).
• Coverage Rate: Proportion of required phases completed ($v_j=1$).
• OrderSafe Score: Binary, plus Critical-Edge Satisfaction (CDS) score:

$$\mathrm{CDS}(D) = \frac{1}{|E_{\mathrm{crit}}|}\sum_{(\phi_i\to\phi_j)\in E_{\mathrm{crit}}}\mathrm{OK}(i\!\to\!j)$$

• Diagnostics: Per-phase precision/recall, Phase-Sequence Agreement (PSA), Attempt-Phase Consistency (APC), review-time, automation rate.

This reporting structure allows clinicians to review only rows where $v_j=0$ or a critical edge fails, significantly streamlining compliance adjudication. Engineers can identify phase-level trends, focusing remediation at a finer granularity than possible with turn-level metrics (Kulkarni et al., 13 Jan 2026).

6. Practical Considerations, Limitations, and Extensions

OIP–SCE’s effectiveness depends on accurate boundary detection for phase transitions, which may involve regular expressions, UI logs, or LLM-based adjudication. In noisy dialogues, boundary ambiguity can result in under- or over-credit for phases. The policy rubrics themselves may drift, requiring periodic recalibration; clinician–engineer collaboration reportedly achieves phase-label $\kappa \approx 0.9$, but maintaining rule validity is ongoing.

Careful integration is required for real-time compliance feedback to avoid alarm fatigue (e.g., live alerts on missed red flags). The phase-graph and compliance logic naturally generalize to other regulated domains: finance (KYC before disclosures), legal (client intake before document sharing), debt collection, and education (consent before data collection), with new rubrics and critical edges defined per domain.

A plausible implication is that OIP–SCE supports both granular model development and routine, safe deployment by ensuring that “all required phases happen, in the right order,” transforming complex regulatory requirements into a tractable, auditable evaluation pipeline (Kulkarni et al., 13 Jan 2026).