STCALIR Framework Overview

• STCALIR is a multi-domain framework that unifies semi-synthetic legal IR, STL inference under covariate shift, and LLM-driven STPA artifact management.
• It employs rigorous pipelines with automation, interpretability, and human-in-the-loop validation to reduce manual efforts and ensure high precision.
• The framework demonstrates significant efficiency gains with measurable performance improvements in retrieval accuracy and artifact linkage effectiveness.

STCALIR Framework

The STCALIR framework denotes a set of technical methodologies, pipelines, or libraries unified under the STCALIR acronym—each distinguished by its application area and scientific goals. Across published research, STCALIR has been defined as: (a) a semi-synthetic test collection framework for low-resource legal information retrieval (Hatem et al., 1 Apr 2026), (b) a conformalized signal temporal logic inference under covariate shift (Wang et al., 28 Mar 2026), and (c) an LLM-integrated framework for the completion, management, and tracing of STPA artifacts (Raeisdanaei et al., 15 Mar 2025). Each instantiation is characterized by a rigorous workflow combining automation, interpretability, and human-in-the-loop protocols for domain-specific evaluation.

1. Semi-Synthetic Test Collections for Legal Information Retrieval

STCALIR, as introduced for Algerian legal IR, provides a pipeline for producing test collections adhering to the Cranfield paradigm, with the core aim of reducing manual annotation by up to 99% (Hatem et al., 1 Apr 2026). The framework targets domains lacking large, curated corpora and gold-standard relevance judgments.

The pipeline consists of five phases:

1. Corpus Acquisition & Preprocessing: Text is extracted via OCR from Algerian legal PDFs, chunked structurally and semantically to conform with input limits, yielding a corpus with $M = 105,\!201$ passages.
2. Topic Creation: Legal experts manually curate $Q = 100$ information-need queries, forming the Cranfield “topics”.
3. Candidate Pool Construction: For each query, retrieval is performed using sparse (BM25) and dense (six fine-tuned bi-encoder) models, and their outputs are merged via Reciprocal Rank Fusion (RRF) with $k=60$, yielding a pool of 1,000 candidates per query.
4. Pseudo-Relevance Judgments: Five cross-encoders score each (document, query) pair in the pool; scores are again fused with RRF. The top 10 are selected as predicted relevant sets $\hat{R}_{10}(q)$.
5. Human Validation: Only these top 10 per query are manually annotated as relevant or non-relevant, yielding the final semi-synthetic label matrix $\hat{Y} = \{y_{q,d}\}$.

This combination delivers a highly efficient annotation protocol, preserving retrieval quality (Hit@10 $\approx$ 0.785) and ensuring system-level ranking concordance with human-annotated benchmarks ($\tau$ = 0.89, $\rho$ = 0.92).

2. Conformalized Signal Temporal Logic Inference under Covariate Shift

The STCALIR framework in the STL setting addresses the model reliability problem under covariate shift between training and deployment (Wang et al., 28 Mar 2026). It aims to learn interpretable STL formulas with guaranteed calibration under distributional shift:

• Problem Statement: Given trajectories $X \in \mathbb{R}^{d\times(T+1)}$ and binary outcomes $Y$, learn an STL formula $Q = 100$0 such that classification is robust to $Q = 100$1, with $Q = 100$2.
• Template-Free Differentiable Inference: Employing TLINet, $Q = 100$3 is trained on $Q = 100$4 by minimizing a margin-based surrogate loss $Q = 100$5.
• Distribution Alignment: Refinement on $Q = 100$6 uses importance weights $Q = 100$7, estimated by k-nearest neighbors in learned embedding space. A Jensen–Rényi divergence regularizer aligns robustness-margin distributions.
• Weighted Conformal Prediction: Post-training, conformal prediction is calibrated with weighted empirical CDFs to guarantee $Q = 100$8 coverage under $Q = 100$9.
• Pseudocode Summary: The pipeline includes STL pretraining, weight estimation, refinement, conformal calibration, and deployment prediction.

The framework provides provable coverage guarantees for the deployed STL predictors, notably under non-exchangeable, shifted data (Wang et al., 28 Mar 2026).

3. LLM-Integrated Completion, Management, and Tracing for STPA

In safety engineering, STCALIR denotes an open-source pipeline for automating the high-volume requirement tracing tasks associated with System-Theoretic Process Analysis (STPA) (Raeisdanaei et al., 15 Mar 2025):

• Artifact Types: Explicit modeling of Losses, Hazards, Unsafe Control Actions (UCAs), Loss Scenarios, each as Pythonic entities.
• Architecture: Modular subsystems include a user interface for artifact definition and review; a model manager for in-memory artifact and linkage graphs; control-structure parsers for draw.io XML diagrams; LLM integration for artifact synthesis via GPT-4o (with vision); and a traceability engine for recommended link discovery across artifacts.
• LLM-Driven Workflow: Prompts for UCA/Loss Scenario synthesis and trace recommendation are designed for high-fidelity batch outputs. Artifacts are deduplicated, mapped to the artifact graph, and reviewed in UI loops.
• Performance Metrics: Artifact Success Rate (useful + correct), F1-score for linkage discovery, and synthesis precision. In empirical validation on canonical STPA case studies, the framework achieved 98% correctness in loss scenario generation and F1 $k=60$0 0.88 for automated linking.
• Implementation: Python 3.10+, OpenAI SDK for LLM calls, networkx for graph modeling, lxml for XML parsing, with all artifacts and trace graphs persisted as dataclasses.

The open library supports CI/CD workflow integration, versioned safety artifacts, and extensibility to new domains and artifact types.

4. Comparative Evaluation and Performance

A unifying feature of STCALIR frameworks in each domain is the substantial reduction in manual effort for high-precision annotation or artifact linkage, validated through quantitative evaluation:

Domain	Annotation Workload Reduction	Retrieval/Linkage Effectiveness	System-level Ranking Concordance
Legal IR (Algeria)	99%	Hit@10 ≈ 0.785	τ = 0.89, ρ = 0.92
STPA Automation	–	UCA Success Rate: 0.78	F1 ≥ 0.88
STL Inference (CovShift)	–	Near-nominal coverage (≥1-α)	–

In each case, STCALIR ensures high reproducibility, human oversight, and in many cases open-source accessibility. The frameworks are evaluated on public datasets (e.g., Mr. TyDi for IR, MIT STPA Handbook cases), and fully document their data flows and performance metrics.

5. Limitations and Open Research Directions

Limitations common across the STCALIR instantiations include:

• Dependence on small human-in-the-loop annotation for quality control (e.g., only top-10 relevance verification per query in IR; human validation of LLM-suggested artifacts in STPA).
• LLM-generated outputs are susceptible to hallucinations (notably ~10% for UCA synthesis), requiring strict deduplication and curation.
• Scalability challenges in quadratic linking algorithms for large-scale artifact graphs.
• Open questions remain regarding case-generalization: real industrial or national-scale systems may pose artifacts or edge-cases unseen in the validating datasets.

Future research directions emphasize full pipeline automation (e.g., end-to-end STPA from incident reports), risk assessment integration, adaptive real-time analyses for safety-critical systems, and large-scale public artifact corpus creation (Raeisdanaei et al., 15 Mar 2025, Hatem et al., 1 Apr 2026).

6. Cross-Domain Generality and Principles

STCALIR exemplifies a general strategy in computational research: the construction of cost-efficient, reproducible, and high-fidelity semi-automated workflows for artifact labeling, model inference, and traceability in domains resistant to large-scale manual annotation. The distinguishing characteristics include multistage retrieval/filtering, model-assisted synthesis augmented by human validation, and the principled fusion of statistical, logical, or symbolic reasoning pipelines. Each framework demonstrates rigorously benchmarked improvements over baseline or manual approaches within its domain (Wang et al., 28 Mar 2026, Hatem et al., 1 Apr 2026, Raeisdanaei et al., 15 Mar 2025).