SCLE: Structured Case-Level Examination

• Structured Case-Level Examination is a systematic method that reviews individual AI cases to reveal rare-event misclassifications and model weaknesses.
• It employs stratified sampling and diagnostic tagging to prioritize high-risk errors and guide targeted model improvements.
• SCLE is applied in domains like pharmacovigilance, cybersecurity, and legal reasoning to enhance transparency and risk governance.

Structured Case-Level Examination (SCLE) is an evaluative and analytical methodology that augments statistical performance metrics with systematic, case-by-case scrutiny of individual AI model outputs. Originating in domains where rare-event recognition imposes critical costs—such as pharmacovigilance, security assurance, and legal reasoning—SCLE provides a rigorous, context-aware framework for understanding, validating, and improving AI systems. By dissecting individual cases, stratifying errors, and tagging diagnostic features, SCLE helps uncover systematic weaknesses and operational risks that aggregate metrics alone cannot reveal.

1. Motivation and Rationale

SCLE was proposed to address inherent limitations in aggregate statistical evaluation (e.g., precision, recall, F₁-score), which may obscure critical patterns, especially when the event of interest is rare and the consequence of error is high (Noren et al., 5 Oct 2025). In domains such as pharmacovigilance, cybersecurity, or rare disease diagnostics, conventional metrics often fail to signal problematic misclassifications or subgroup-specific errors due to class imbalance. Structured case-level review, by contrast, enables direct appraisal of real individual predictions, illuminating not only performance gaps but also reasons for success and failure.

Key motivations:

• Identification of systematic misclassification, especially those affecting critical subgroups.
• Uncovering rare but unacceptable “never events,” which bear disproportionate operational and reputational costs.
• Explaining model performance in context, enabling actionable feedback for real-world deployment.

2. Methodological Framework

The core SCLE process unfolds in several stratified, human-in-the-loop steps (Noren et al., 5 Oct 2025):

1. Case Stratification: Predictions are partitioned into three principal categories:
   • True Positives (TP): correctly identified rare events
   • False Positives (FP): incorrect identification of an event
   • False Negatives (FN): missed identification of true events

True negatives are de-emphasized, as their high abundance imparts little diagnostic value in rare-event settings.

1. Sampling Based on Risk and Confidence: Within each stratum, cases may be further subdivided by distance to the decision boundary or assigned risk level. This ensures review effort is focused on both “confident errors” and borderline cases. Sample sizes may be set according to operational priorities—e.g., expanded review of “never events.”
2. Diagnostic Tagging: Human reviewers assign structured tags to each examined case, such as:
   • “Never event”: an error that must never occur
   • “Unexpected error,” “Input data issue,” “Test set issue,” or “Triviality” These tags form a basis for feedback loops, targeted retraining, and refinement of annotation guidelines.
3. Integration and Feedback: Insights from SCLE are used to guide model improvement at multiple levels:
   • Adjusting thresholds or cost-sensitive targets
   • Revising data pipelines or annotation schemas
   • Benchmarking against state-of-the-art and operational standards

Step	Action	Outcome
Stratify Predictions	TP, FP, FN	Focused error analysis
Sample by Risk	Boundary/confidence	Prioritized review effort
Diagnostic Tagging	Label cases	Systematic error classification
Feedback Integration	Threshold/tuning	Actionable model improvements

3. Applications Across Domains

Pharmacovigilance is a primary exemplar for SCLE. The methodology was instantiated across three studies (Noren et al., 5 Oct 2025):

• Rule-based report retrieval: Case-level review exposed systematic FNs caused by mapping errors (e.g., unmapped ICD codes and incomplete term lists).
• Probabilistic duplicate detection: SCLE revealed performance contrasts between models, enabling granular benchmarking of classification boundaries.
• Automated redaction: Comprehensive FN/FP review, with corresponding diagnostic tags, surfaced privacy risks (e.g., leaked names) and adverse effects due to over-masking.

The principles are generalizable:

• Fraud detection: Rare fraud cases require focused analysis of misclassifications to prevent operational loss.
• Medical diagnostics: Rare disease prediction systems benefit from detailed case-level review to ensure clinical robustness.
• Cybersecurity: “Never events” (e.g., undetected breaches) can be prioritized for exhaustive error review.

4. Complementarity to Statistical Evaluation

SCLE does not supplant but complements classical performance metrics. While aggregate measures (such as $F_1 = \frac{2pr}{p+r}$) remain vital for benchmarking, SCLE provides context and explanation, surfacing:

• Systematic errors hidden by class prevalence skews
• Subpopulation vulnerabilities (underserved groups)
• Meaningful operational insights for risk management

SCLE’s review of both errors and successes helps organizations interpret why errors persist, how improvements can target root causes, and to what extent performance metrics correspond to real operational value. Cost-sensitive targets are used to realign quantitative performance with domain-specific requirements.

5. Relationship to Structured Argumentation and Assurance

SCLE’s emergence parallels structured argumentation practices in domains such as security assurance (Mohamad et al., 2020) and legal reasoning (Araszkiewicz, 2024). In SACs (Security Assurance Cases), structured models (e.g., Goal Structuring Notation, Claim-Argument-Evidence frameworks) are used to trace security claims to supporting evidence. SCLE may incorporate these argument patterns, evidence structuring, and documentation processes to bolster the rigor of case-level review. Similarly, in statutory interpretation, case frames formalize interpretative arguments, which can enrich SCLE tagging schemes or error context.

Concretely, the integration can take the following forms:

• Linking errors to structured argument nodes (claims, evidence)
• Contextualizing diagnostic tags using assurance case hierarchies
• Feeding SCLE outputs into iterative SAC/SCLE assessment workflows

6. Ongoing Monitoring and Risk Governance

SCLE is envisioned as a continuous process, not a one-off validation. As AI systems evolve or as data drift arises, repeated stratified case-level review ensures sustained performance and documented risk governance. This process is fundamental in regulated domains, where transparency and accountability for decision errors are mandated.

The methodology promotes:

• Regular sampling and stratified case audits
• Documentation of observed error patterns and operational impacts
• Adjustment strategies based on evolving real-world feedback

This continuous feedback closes the loop between development, monitoring, and deployment.

7. Limitations and Future Directions

SCLE faces certain challenges:

• Scalability: Routine manual review of large case volumes requires judicious sampling, automated prioritization, or collaborative review platforms.
• Tool support: Current platforms lack mature, integrated toolchains for SCLE; future work (as in the security assurance case domain) may focus on unified environments tracing dependencies, evidence, and review tags (Mohamad et al., 2020).
• Standardization: Diagnostic tags and stratification schemes should converge to domain-wide standards for comparability and accountability.

Potential future developments include:

• Automated prioritization of review samples by risk scoring
• Integration of SCLE tagging with assurance case or legal reasoning frameworks
• Extension of SCLE to novel rare-event domains, incorporating cost-sensitive error management and structured documentation

Summary

Structured Case-Level Examination (SCLE) constitutes a rigorous, stratified methodology for AI appraisal in rare-event and high-stakes decision domains. By focusing on direct review and tagging of individual cases—especially within critical error strata—SCLE reveals operational risks and systematic weaknesses unaddressed by aggregate metrics alone. Its principles are extensible to multiple domains and align closely with structured argumentation frameworks, supporting continuous risk governance and transparent model improvement (Noren et al., 5 Oct 2025, Mohamad et al., 2020, Araszkiewicz, 2024).