- The paper introduces SR4CS, a corpus of 1,212 systematic reviews in computer science, each with original Boolean queries and resolved references, providing a valuable resource for evaluating automated retrieval methods.
- The study demonstrates that approximated expert Boolean queries show the highest precision (0.352) while dense retrieval methods exhibit superior ranking metrics and recall (e.g., 0.281 MAP and 0.309 F₃), highlighting the effectiveness of different retrieval strategies.
- The research reveals that zero-shot Boolean generation, despite off-the-shelf models, does not provide competitive results in terms of precision and recall without specialized iterative methods or more intelligent prompting, hence the need for domain-specific improvements or feedback-driven query construction.
Motivation and contribution
Systematic reviews are the standard instrument for evidence synthesis, and their retrieval phase—typically realized through expert-designed Boolean queries—directly determines the pool of eligible studies, since recall lost at retrieval cannot be recovered downstream. Although automation of query formulation, screening, and synthesis has attracted substantial attention, evaluation resources remain concentrated in biomedicine, with test collections such as the SIGIR 2017 SysRev Query Collection (94 Cochrane topics over ~26M MEDLINE records), CLEF TAR 2019 (123 topics over PubMed), and the Seed Studies Collection (40 medical topics) (2604.16330). No comparable large-scale resource exists for computer science.
The paper introduces SR4CS, a corpus of 1,212 systematic reviews in computer science, each paired with its original expert-designed Boolean search strategy, an approximated normalized variant restricted to titles and abstracts, curated reference pools, and structured methodological metadata (databases searched, inclusion/exclusion criteria, temporal and language restrictions, research questions, snowballing status). The collection contains 104,316 resolved references, of which 89,447 include abstracts; reviews average ~83 references each (median 74, range 4–459). The resource is released openly on Zenodo with documentation and code, positioning it as the first domain-specific benchmark of this scale outside biomedicine.
Corpus construction
The pipeline comprises three stages. In data collection, candidate systematic reviews were retrieved and filtered to those explicitly reporting Boolean queries. For parsing, PDFs were converted to semantically enriched Markdown using Nanonets-OCR-s, a vision–language OCR model chosen because Boolean queries frequently appear in figures or tables that plain-text OCR mishandles. Structured field extraction was then performed zero-shot with GPT-4.1 Mini across nine field categories.
Extraction quality was assessed by manual inspection plus independent manual extraction on a random 10% sample: fields matched in 85% of cases at the field level, with errors limited to wording discrepancies in inclusion/exclusion criteria or values inferred from indirect clues rather than explicit statements. This supports zero-shot LLM extraction as a viable basis for large-scale corpus construction, though the authors acknowledge it leaves room for feedback-driven, more robust extraction methods.
Reference resolution used a hybrid AnyStyle + GROBID pipeline, a configuration previously shown effective for citation analysis. Non-scientific items (reports, interviews, patents) were removed (10,101 entries excluded), and remaining references were enriched with abstracts from Crossref, OpenAlex, Semantic Scholar, PubMed, Europe PMC, and the arXiv API. Resolution prioritized DOI matches, falling back to high-threshold title–author matching with publication-year consistency checks to limit false positives. Scopus (649 reviews), ScienceDirect (478), ACM DL (439), Web of Science (418), IEEE Xplore (387), and Google Scholar (346) dominate as cited sources—consistent with computer science review practice.
Baseline experiments
To illustrate intended use, the authors evaluate four methods under a unified title-and-abstract-only setting, capped at 1,000 documents per query for non-Boolean methods:
- Expert Boolean (approx.): original queries rewritten into SQLite FTS5 MATCH syntax via GPT-4.1-mini, preserving structure and stripping unsupported metadata filters. Manual verification of a 25% sample required corrections to only seven queries.
- Zero-shot LLM Boolean: GPT-4.1-mini-generated queries from review title and objective.
- BM25 and dense retrieval (all-MiniLM-L6-v2) using title + objective concatenation.
Evaluation against curated reference pools yields macro-averaged precision, recall, F₁, F₃, MAP, P@10, and R@100:
| Method |
Precision |
Recall |
F₁ |
F₃ |
MAP |
P@10 |
R@100 |
| Expert Boolean (approx.) |
0.352 |
0.342 |
0.224 |
0.241 |
0.173 |
0.490 |
0.238 |
| GPT-4.1-Mini Boolean (ZS) |
0.298 |
0.099 |
0.095 |
0.085 |
0.054 |
0.256 |
0.083 |
| BM25 (Title+Obj) |
0.041 |
0.512 |
0.074 |
0.223 |
0.175 |
0.432 |
0.262 |
| Dense (MiniLM) |
0.057 |
0.702 |
0.104 |
0.309 |
0.281 |
0.545 |
0.373 |
Three findings stand out. First, approximated expert Boolean queries achieve the highest precision (0.352), confirming Boolean logic as the most effective means of controlling screening effort; their moderate recall (0.342) is attributed primarily to the approximation itself—which strips database-specific operators, controlled vocabularies, and citation-based expansion—rather than deficiencies of the original strategies. Second, dense retrieval dominates ranking metrics (MAP 0.281, P@10 0.545, R@100 0.373) and delivers recall of 0.702, demonstrating robustness to vocabulary mismatch under metadata-constrained retrieval. Third, the paper reports a clear negative result: naive zero-shot Boolean generation with an off-the-shelf model fails badly (recall 0.099), recovering neither the coverage nor the selectivity of expert strategies—an explicit indictment of unstructured prompt-based query generation and an argument for adaptive, feedback-driven query construction.
Limitations and open questions
Several caveats bear directly on interpretation. The expert Boolean baseline is an approximation: normalization removes database-specific syntax, MeSH-style controlled vocabularies, and citation chasing, so the reported recall of 0.342 understates what the original strategies achieve against full bibliographic databases. Reference pools derive from the reviews' own reported included studies rather than exhaustive re-execution of searches, so absolute recall figures are bounded by the completeness of the source reviews. Extraction relies on zero-shot LLM processing validated only on a 10% sample at 85% field-level agreement, leaving residual noise in metadata fields. Ranking evaluation of Boolean result sets uses SQLite's internal BM25 ordering—a deterministic but somewhat artificial choice for an inherently unranked paradigm. Open questions include whether agentic, feedback-grounded extraction can improve field fidelity, whether the workflow transfers to domains such as education or social science, and how retrieval methods behave when integrated with open scholarly infrastructures such as OpenAlex for end-to-end evaluation.
Conclusion
SR4CS fills a concrete resource gap by providing the first large-scale, openly licensed test collection of systematic reviews outside biomedicine, complete with original and normalized Boolean strategies, resolved reference pools, and methodological metadata. Its baseline experiments establish reproducible reference points that expose systematic precision–recall trade-offs across Boolean, lexical, and dense retrieval paradigms, and demonstrate that current zero-shot LLM Boolean generation is not yet competitive with expert-designed strategies.