- The paper introduces MetaSyn, a 442-paper benchmark with expert-reviewed PI/ECO criteria, hard negatives, and nine metrics covering retrieval, screening, and synthesis across the full meta-analysis pipeline.
- LLM retrievers achieve up to 90.9% Recall@200, but the best end-to-end system recovers only 52.7% of eligible studies, showing that eligibility screening—not literature retrieval—is the primary bottleneck.
- The results reveal distinct system trade-offs in recall, precision, criteria adherence, and conclusion accuracy, emphasizing the need for protocol-aware screening, multi-query retrieval, and stage-specific evaluation rather than a single aggregate score.
Overview
"MetaSyn" is a benchmark for evaluating LLM agents on the full meta-analysis pipeline: literature retrieval, PI/ECO-guided screening, and synthesis (2606.17041). The authors argue that existing scientific-synthesis benchmarks evaluate topical relevance or narrative coherence, but not the criterion-based exclusion logic that makes a systematic review trustworthy. A study may be topically indistinguishable from a ground-truth inclusion yet fail a single eligibility component (population, comparator, design, follow-up window), and no prior benchmark provides verifiable ground truth across the entire retrieval–screening–synthesis chain. MetaSyn addresses this by exploiting the fact that published meta-analyses already enumerate their included studies, state their eligibility criteria, and report their conclusion direction, so stage-level supervision can be inherited from the source papers without independent relabeling.
Dataset construction and composition
The dataset is drawn from the Nature Portfolio: from an initial pool of 34,375 candidate articles (Scientific Reports was excluded to keep the editorial baseline uniform), roughly 50 human annotators applied a four-stage protocol—accessibility verification, confirmation of genuine quantitative meta-analysis, extraction of analyzed-study lists from supplementary tables and forest plots, and metadata extraction. The filtering is deliberately selective, yielding 442 meta-analyses with complete, extractable ground truth. PI/ECO structuring of research questions uses GLM-4.6 for an initial pass, but every field is human-reviewed and corrected.
Each instance is paired with a PubMed-anchored corpus of 140,585 articles: 8,674 corpus-matched positives (studies actually analyzed in at least one source meta-analysis) and 131,911 hard negatives retrieved by title similarity—topically close but failing at least one PI/ECO criterion, at roughly a 1:15 positive-to-negative ratio. Domain coverage is broad: clinical specialties account for 67.6% (oncology 16.5%, mental health 14.3%), with the remaining 32.4% spanning digital health, social sciences, and environmental science. The test split comprises 88 held-out queries; the 354-paper training split supports retriever fine-tuning.
Two structural caveats bear on all reported numbers. Because systematic searches draw on EMBASE, Cochrane Central, and trial registries in addition to PubMed, only 45.7% of reported included studies are corpus-matched on average (66.4% on the test split), so all metrics are computed against the PubMed-matched subset. The authors note this makes reported scores conservative lower bounds for multi-source pipelines, and that domain-stratified evaluation is left unaddressed.
Benchmark design
MetaSyn defines two tasks. The end-to-end task gives a system the research question and PI/ECO elements and requires a structured report: included studies, a conclusion, and key insights. The isolated retrieval task withholds search strategies and date bounds so the system must formulate its own queries, with Recall@K against corpus-matched ground truth; articles published after the original search end date (99.3% coverage) are excluded to prevent temporal leakage. Nine metrics span three dimensions—inclusion quality (Inc.R, Inc.P, Inc.F1, Scr.A), criteria adherence (Inc.C, Exc.C), and synthesis (Dir.A, Insights, SQ). Four are exact set comparisons; the five evaluator-dependent metrics are validated against 8-annotator pairwise judgments.
Retrieval results: three stacking levels of signal
The isolated retrieval experiments establish a clear ceiling. BM25 reaches 65.4% Recall@100; off-the-shelf BGE dense retrieval adds +12.8 points (78.2%, p<10−9); MA-Retriever, BGE fine-tuned on the MetaSyn training split with MultipleNegativesRankingLoss, adds a further +5.4 points to 83.7%, and lifts Recall@200 to 90.9% (p<10−5). The gains stack rather than substitute, indicating the training split carries eligibility signal beyond generic semantic pretraining. Fine-tuning helps most on large meta-analyses (+7.8 points R@100 for those with 50+ included studies), the regime where target sets are topically diverse but methodologically coherent. Four test queries remain below R@100 = 0.3 under all three retrievers, spanning network meta-analyses, tissue-specific vocabularies, and heterogeneous study designs—cases where a single query cannot express the protocol's cross-cutting constraints. At K=200, each retrieved pool contains on average 16 true positives among 184 distractors, a 1:11 screening ratio.
End-to-end results: a screening bottleneck
Twelve configurations—nine RAG variants across DeepSeek-R1, GLM-5, and GPT-5 with three retrievers, plus ProtoMA, a protocol-faithful GPT-5 agent with batched PI/ECO screening and per-article extraction—produce the central finding: no system recovers more than 52.7% of ground-truth included studies, despite the 90.9% retrieval ceiling. The ~38-point gap is attributable almost entirely to screening, not retrieval.
| System |
Retrieval |
Inc.R |
Inc.P |
Inc.F1 |
Scr.A |
Dir.A |
| RAG (DeepSeek-R1) |
MA-Retriever |
15.6 |
16.1 |
11.6 |
84.1 |
56.8 |
| RAG (GLM-5) |
MA-Retriever |
52.7 |
26.6 |
31.0 |
73.9 |
61.4 |
| RAG (GPT-5) |
BM25 |
42.5 |
36.1 |
35.0 |
87.4 |
50.0 |
| ProtoMA (GPT-5) |
MA-Retriever |
35.6 |
55.5 |
39.8 |
93.7 |
31.8 |
The four system families occupy distinct operating profiles rather than a single ranking. DeepSeek-R1 cites fewer than five studies per query, leaving no channel for retrieval improvements to propagate (Inc.R gains within noise). GLM-5 accepts a large share of the pool, tracking retrieval ceilings closely (44.1% → 52.7%, p<0.001) but accumulating false inclusions. GPT-5 exhibits a counterintuitive inversion: its Inc.R and Inc.P both decline as retrieval quality rises (42.5% → 31.7% under dense retrieval, both p<0.001). The authors argue a selective-screening account would predict a precision–recall tradeoff; the joint drop instead implicates pool composition—denser retrieval substitutes lexically obvious positives with topically proximate PI/ECO-failing candidates that GPT-5 then includes. ProtoMA achieves the precision extreme (Inc.P 55.5%, Scr.A 93.7%) but the lowest Dir.A, partly mechanical: its hedged, uncertainty-explicit conclusions map to "Mixed" on 80.7% of papers, so its Dir.A is a lower bound under the categorical metric. No system ranks first on all dimensions; the authors contend a single aggregate score would fold four distinct failure modes into a metric-weighting artifact.
Metric validation
The five evaluator-dependent metrics were validated on 30 papers with 390 pairwise tasks rated by 8 annotators. Dir.A is the standout: 93.5% annotator–annotator agreement, 91.2% annotator–LLM label agreement, and Spearman ρ=+0.82—the only metric supporting fine-grained system ranking. Exc.C (ρ=+0.59) and Inc.C (ρ=+0.24) and Insights support only coarse, cross-regime contrasts; SQ serves as a diagnostic descriptor. The experimental claims in the paper are accordingly anchored on the ID-based metrics and Dir.A. A limitation the authors concede: the categorical Dir.A metric systematically penalizes ProtoMA's uncertainty-explicit outputs, and a certainty-aware variant is deferred to future metric work.
Limitations and open questions
Two scope restrictions are explicit. First, the PubMed-only corpus means systems retrieving valid non-PubMed evidence receive no credit, and the corpus-match rate (45.7% overall) varies by domain—clinical fields that draw on EMBASE and registries are underrepresented in ground truth, so aggregate retrieval numbers likely understate domain-level variation. Second, the Dir.A metric conflates hedging with directional error, as noted above. Beyond these, the residual retrieval failures on four queries raise the open question of how multi-query or protocol-aware query formulation should handle eligibility constraints that a single query cannot express, and the GPT-5 pool-composition sensitivity leaves open how screening components can be made robust to semantically diverse rather than lexically aligned candidate pools.
Conclusion
MetaSyn contributes a 442-paper, expert-curated benchmark with stage-level ground truth inherited structurally from published meta-analyses, a 140k-article corpus with hard negatives, and nine stage-attributed metrics validated against human judgments. Its principal empirical result quantifies a screening bottleneck: with 90.9% of eligible studies retrievable at K=200, the best end-to-end system still recovers only 52.7%, because current LLMs cannot reliably apply PI/ECO criteria to pools of topically equivalent candidates. The stage-attributed evaluation design—rather than any single aggregate score—is what makes this diagnosis, and the per-system improvement targets that follow from it, legible.
