A Large-Scale, Cross-Disciplinary Corpus of Systematic Reviews

Abstract: Existing benchmarks for systematic reviewing remain limited either in scale or in disciplinary coverage, with some collections comprising only a modest number of topics and others focusing primarily on biomedical research. We present Webis-SR4ALL-26, a large-scale, cross-disciplinary corpus of 301,871 systematic reviews spanning all scientific fields as covered by OpenAlex. Using a multi-stage pre-processing pipeline, we link reviews to resolved OpenAlex metadata and reference lists and extract, when explicitly reported, structured method artifacts relevant to retrieval and screening. These artifacts include reported search strategies (Boolean queries or keyword lists) that we normalize into executable approximations, as well as reported inclusion and exclusion criteria. Together, these layers support cross-domain benchmarking of retrieval and screening components against review reference lists, training and evaluation of extraction methods for review artifacts, and comparative meta-science analyses of systematic review practices across disciplines and time. To demonstrate one concrete use case, we report large-scale baseline retrieval signals by executing normalized search strategies in OpenAlex and comparing retrieved sets to resolved reference lists. We release the corpus and the pre-processing pipeline, along with code used for extraction validation and the retrieval demonstration.

• The paper introduces a large-scale corpus of 301,871 systematic reviews from 27 domains to address disciplinary coverage gaps in review benchmarks.
• The methodology employs a three-stage enrichment pipeline using metadata retrieval, full-text acquisition, and LLM-based structured artifact extraction.
• Results demonstrate effective normalization of queries and precise extraction of methodological artifacts, supporting advanced retrieval and meta-science analyses.

A Large-Scale, Cross-Disciplinary Corpus of Systematic Reviews

Motivation and Overview

Systematic reviews are central to evidence synthesis across scientific domains, serving as a methodological foundation for transparent, reproducible literature aggregation with explicit eligibility criteria. Existing evaluation resources for systematic review automation are predominantly restricted to biomedicine and limited in scale and diversity. The paper introduces Webis-SR4ALL-26 (2604.22864), a novel corpus of 301,871 systematic reviews from OpenAlex, spanning 27 scientific domains and integrating metadata, resolved reference lists, and structured methodological information extracted from full texts. This corpus systematically addresses coverage gaps in retrieval and screening benchmarks for systematic reviews, supporting cross-domain method benchmarking, artifact extraction, and meta-science analyses.

Corpus Construction and Content

Data Collection and Deduplication

The corpus construction employs a three-stage enrichment pipeline: metadata retrieval from OpenAlex, full-text acquisition, and structured information extraction. Candidate reviews are identified using title-based heuristics targeting explicit systematic review declarations, removing duplicates via DOI/OpenAlexID/title normalization. Inclusion criteria involve English language, valid DOI, explicit methodology (excluding updates and non-original reviews), resulting in 280,886 OpenAlex-derived reviews. Additional coverage is achieved through integration of major benchmark datasets (CLEF TAR, SysRev-Query, SysRev-Seed, CSMeD, AutoBool), yielding 301,871 records post-deduplication.

Cross-Disciplinary Scope

Domain assignment via OpenAlex metadata reveals dominant contributions from Medicine (64%), Psychology (6%), Health Professions (5%), Biochemistry/Genetics/Molecular Biology (4%), Social Science (3%), and Neuroscience (3%). The remaining disciplines include Dentistry, Environmental Science, Engineering, Computer Science, Immunology/Microbiology, and Economics, accentuating broad applicability and methodological diversity. All records are citation-linked through OpenAlex, establishing a reproducible reference backbone for benchmarking.

Full-Text Acquisition and Parsing

Full texts are obtained for 72,678 records using OpenAlex-supplied PDF links. Parsing utilizes PaddleOCR-VL, a vision-LLM optimized for scientific document layouts, achieving markdown conversion with competitive effectiveness and efficiency. This facilitates subsequent information extraction for methodological fields.

Information Extraction Pipeline

The corpus is enriched via an LLM-based pipeline targeting extraction of methodological artifacts: study objectives, research questions, search strategies, eligibility criteria, study counts, date ranges, databases, and citation chasing approaches.

Grounded Extraction and Verification

A verify-then-repair paradigm is adopted for artifact extraction, leveraging Qwen3-32B for primary extraction from OCR-parsed full texts, with explicit evidence spans copied from PDFs. Two-stage validation involves string-level OCR evidence anchoring and semantic verification via MiniCheck [2024.emnlp-main.499]. Extractions failing either validation are discarded; targeted repair prompts address incomplete fields, maintaining high precision and verifiability.

Figure 1: Grounded extraction of review key information. Verbatim source spans from the PDF are transformed into structured schema fields via the verify-then-repair pipeline.

Evidence-centric structured representations are normalized where possible, e.g., temporal constraints mapped to year-range. Analysis of field coverage shows methodological artifacts are available for 10–20% of reviews with full texts, with 12.7% comprising the "effectively usable" subset (objective, search strategy, eligibility criteria present).

Qualitative Extraction Analysis

Manual review of extraction quality across outcome classes (filled, partial, null) confirms strong performance for explicit, well-structured reviews; failures typically arise from document misalignment (errata/supplements) or distributed/implicit information exceeding strict verification parameters. The pipeline prioritizes precision for verifiable fields, potentially at the expense of recall on loosely structured artifacts.

Retrieval Demonstration and Baseline Signals

Reported search strategies (Boolean expressions or keyword lists) are normalized to canonical Boolean queries for OpenAlex execution, stripping database-specific syntax and external constraints. Normalization applies instruction-tuned Qwen3-32B with rule-based postprocessing, only retaining queries with sufficient topical terms and valid logical structure. Null assignment occurs when normalization cannot yield meaningful executable queries.

Queries are executed against OpenAlex API in bucketed result bands; retrieval signals align returned sets with review reference lists, providing metrics for recall, precision, and $F_1$/$F_3$ scores. Results indicate:

• Normalized Boolean strategies achieve recall = 0.245, precision = 0.014, $F_1$ = 0.019, outperforming keyword-only queries (recall = 0.180, precision = 0.016, $F_1$ = 0.018).
• Precision remains low, consistent with the broad scope and abstraction of normalized queries, but relative recall differences provide actionable benchmarking signals.

Implications and Limitations

Webis-SR4ALL-26 establishes a scalable, open infrastructure for benchmarking retrieval and screening automation in systematic reviews beyond biomedicine. The explicit mapping to OpenAlex allows reproducible, cross-domain evaluation and artifact extraction, with the structured methodology fields enabling training of advanced LLM-based extraction models. The corpus supports meta-research on systematic review reporting practices, the evolution of search strategies, and discipline-specific methodological adaptation.

Several design limitations are noted:

• Title-based identification favors precision but omits reviews without explicit systematic review nomenclature.
• Full-text availability is incomplete; extraction coverage is uneven and emphasizes verifiable artifacts.
• Verification protocol prioritizes precision, potentially excluding implicit or distributed fields.
• Reference sets resolve cited works, which may diverge from included-study gold standards due to background citations.
• Query normalization abstracts from database-specific search, limiting direct benchmarking of expert-crafted retrieval performance.

Future Directions

Prospective work may integrate broader metadata and full-text signals for systematic review identification, expand full-text coverage, reconstruct included-study sets for more rigorous screening evaluation, and develop extraction paradigms balancing recall and verifiability. The resource facilitates continued progress in information retrieval, LLM-driven automation, and science-of-science investigations.

Conclusion

Webis-SR4ALL-26 (2604.22864) constitutes a foundational cross-disciplinary corpus for systematic reviews, enabling scalable, reproducible benchmarking and research on retrieval, screening, and artifact extraction. By unifying structured methodology with citation-backed bibliography from OpenAlex, the corpus closes longstanding disciplinary coverage gaps, supports LLM training and evaluation, and advances comparative analyses of systematic review practices. The work lays groundwork for future enhancements in systematic review automation, cross-domain evidence synthesis, and science policy analysis.