STCALIR: Semi-Synthetic Test Collection for Algerian Legal Information Retrieval

Abstract: Test collections are essential for evaluating retrieval and re-ranking models. However, constructing such collections is challenging due to the high cost of manual annotation, particularly in specialized domains like Algerian legal texts, where high-quality corpora and relevance judgments are scarce. To address this limitation, we propose STCALIR, a framework for generating semi-synthetic test collections directly from raw legal documents. The pipeline follows the Cranfield paradigm, maintaining its core components of topics, corpus, and relevance judgments, while significantly reducing manual effort through automated multi-stage retrieval and filtering, achieving a 99% reduction in annotation workload. We validate STCALIR using the Mr. TyDi benchmark, demonstrating that the resulting semi-synthetic relevance judgments yield retrieval effectiveness comparable to human-annotated evaluations (Hit@10 \approx 0.785). Furthermore, system-level rankings derived from these labels exhibit strong concordance with human-based evaluations, as measured by Kendall's τ (0.89) and Spearman's \r{ho} (0.92). Overall, STCALIR offers a reproducible and cost-efficient solution for constructing reliable test collections in low-resource legal domains.

• The paper presents a semi-synthetic framework that integrates automated retrieval, advanced pooling, and expert validation to construct a robust Algerian legal test collection.
• It employs a dual chunking strategy and combines BM25 with transformer-based models for effective candidate retrieval and ranking, achieving high concordance with human annotations.
• The framework reduces expert annotation by 99% while maintaining strong evaluation metrics, offering a cost-effective solution for low-resource legal IR domains.

STCALIR: A Semi-Synthetic Framework for Test Collection Construction in Algerian Legal Information Retrieval

Introduction

The paper "STCALIR: Semi-Synthetic Test Collection for Algerian Legal Information Retrieval" (2604.00731) presents a comprehensive framework that addresses a crucial bottleneck in Information Retrieval (IR) evaluation for understudied and low-resource domains. Specifically, the authors focus on legal texts in Algerian Arabic—a resource-scarce environment with significant complexity in corpus acquisition, topic construction, and relevance judgment labeling.

Traditional Cranfield-style test collections, which form the backbone of retrieval evaluation, require immense manual effort in corpus curation and annotation, especially for high-stakes domains such as law. The STCALIR framework blends automated retrieval, advanced multi-system pooling, and minimal expert validation to yield a semi-synthetic collection that is both cost-efficient and empirically robust.

Methodological Overview

The STCALIR pipeline is defined as a five-phase process, systematically designed to minimize required manual labor while retaining evaluation integrity. The methodology is engineered to function fully with open-source models, sidestepping the dependence on proprietary LLMs and circumventing the prohibitive expense of large-scale annotation campaigns.

Figure 1: Five-phase methodology of the STCALIR pipeline, from corpus extraction to human-validated relevance judgments.

Corpus Acquisition and Preprocessing

The source material consists of Algerian Arabic legal documents extracted from government PDFs. Given the document heterogeneity and noisy OCR outputs, the pipeline enforces a dual chunking strategy: initial rule-based segmentation at the article level, followed by embedding-based semantic chunking for granular, model-compatible text units.

Figure 2: Original Algerian Arabic legal text and its precise English translation, exemplifying the nature of the targeted legal corpus.

Topic Generation and Annotation

Expert-generated queries are treated as essential for capturing nuanced information needs in the legal domain. The annotation interface ensures query quality and diversity.

Figure 3: Web-based annotation tool for systematic generation and management of legal information needs as retrieval topics.

Multi-Stage Retrieval and Pooling

Initial candidate retrieval employs both BM25 and a panel of bi-encoder models, drawing 1,000 candidates per topic, fused via Reciprocal Rank Fusion (RRF) to maximize coverage of potential relevant passages. Re-ranking is performed by an ensemble of cross-encoders, with further RRF aggregation to identify top-10 passages per topic—drastically limiting the scope for subsequent expert review.

Human Validation

The final relevance labeling is applied only to the system-prioritized top 10 results per query, mediated by a tailored web interface.

Figure 4: Web interface for manual relevance assessment by domain experts.

Experimental Protocol and Numerical Results

The STCALIR framework is validated on the Mr. TyDi Arabic benchmark and the newly constructed STCALIR dataset, comparing performance against ground-truth human annotations and evaluating retrieval effectiveness and system ranking concordance.

Key findings include:

• Hit@10 alignment: The semi-synthetic judgments yield a Hit@10 of approximately 0.785 (STCALIR pipeline) on benchmark datasets, closely tracking the fully manual human labels.
• Ranking correlation: System-level rankings induced by semi-synthetic and human labels exhibit Kendall’s $\tau = 0.89$ and Spearman’s $\rho = 0.92$, confirming near-perfect concordance.
• Annotation reduction: The expert annotation workload is reduced by 99%, owing to aggressive multi-stage pre-filtering with minimal impact on retrieval fidelity.
• Model-agnosticity: The pipeline showcases robust cross-domain generalization, with model selection driven by empirical generalization ratios rather than in-domain overfitting.

  Figure 5: High system-level ranking correlation between human and semi-synthetic judgments ($\text{MRR}@10$), evidencing the reliability of the STCALIR methodology.

Discussion

Efficiency-Efficacy Tradeoff

The staged retrieval, pooling, and re-ranking architecture of STCALIR allows aggressive candidate pruning without significant sacrifices in recall or ranking quality. Empirical analysis demonstrates that most relevant documents are consistently elevated within the top ranks, and further expansion of cross-encoder ensembles yields diminishing returns for $\text{Hit}@10$.

Practical Advantages

• Reproducibility: Open-source dependency and modest hardware requirements enable ready adoption in resource-constrained academic contexts.
• Domain Adaptation: All core models can be fine-tuned for legal-domain specificity, and the framework flexibly generalizes to other low-resource IR environments.
• Transparency and Risk Mitigation: The methodology avoids the hallucination risks associated with generative LLMs, critical in legal or sensitive text processing.

Limitations and Future Work

• Corpus Quality: OCR and segmentation errors present ongoing challenges, with improvements in Arabic OCR directly translating to better corpus fidelity.
• Model Dependence: Retrieval effectiveness is coupled to the adaptation and performance of transformer encoders and may benefit from lightweight in-domain fine-tuning strategies (e.g., LoRA).
• Generalizability: While strong, validation is limited to monolingual Arabic legal texts; future work can extend the framework to multilingual or other specialized legal subdomains.

Implications for IR and AI Research

The robust empirical findings demonstrate that semi-synthetic test collections can act as reliable proxies for fully human-annotated resources in system evaluation. The framework's design principles—systematic pooling, aggressive automation, expert-in-the-loop curation, and architectural modularity—set a template for broader test collection construction in data-scarce domains, including medical, financial, or vernacular legal corpora.

On the theoretical front, the strong system-level consistency metrics suggest that for comparative system development and benchmarking, semi-synthetic resources produced via such systematic pipelines may suffice, provided that the pooling strategy achieves adequate recall and the final labels are subjected to strict expert review.

Conclusion

STCALIR defines a rigorously validated, open, and reproducible pipeline for scaling test collection construction in low-resource settings, achieving strong agreement with manual benchmarks and massive reductions in annotation effort. Its applicability extends beyond Algerian legal documents, serving as a model for efficient evaluation resource construction in emerging, specialized, and data-constrained IR domains.