AutoSynthesis: An agentic system for automated meta-analysis

Abstract: Evidence synthesis is crucial for turning primary research into reliable knowledge for science, medicine, education, and policy. Yet, quantitative evidence synthesis remains largely manual and difficult to scale. Here, we introduce AutoSynthesis, an end-to-end multi-agent system for automated meta-analysis. Given a research question in natural language, AutoSynthesis formulates a search strategy, retrieves scientific literature, screens candidate studies, assesses full-text eligibility, extracts quantitative statistics, computes standardized effect sizes, and finally performs random-effects meta-analysis. AutoSynthesis further supports heterogeneity analysis to examine how effect sizes vary across moderators, as well as risk-of-bias assessment. As output, AutoSynthesis produces a transparent report aligned with PRISMA guidelines. In our application, AutoSynthesis screened over 28 studies and extracted more than 20 quantitative claims. The pooled effect estimates produced by AutoSynthesis are similar to Hedges' $g$ of expert-conducted meta-analyses, indicating close agreement with manual evidence synthesis. Together, these results show that AutoSynthesis can make quantitative evidence synthesis more scalable, thereby supporting evidence-based decision-making across disciplines.

• The paper introduces AutoSynthesis, an agentic system that automates the entire meta-analysis workflow from literature search to PRISMA-compliant reporting.
• It details a modular multi-agent architecture that performs structured protocol generation, data extraction, and effect size computation with robust error correction.
• Empirical evaluations show strong correspondence with expert benchmarks, demonstrating its potential for scalable and rapid evidence synthesis.

AutoSynthesis: An Agentic System for Automated Meta-Analysis

Introduction and Motivation

Quantitative evidence synthesis through meta-analysis is central to producing reliable, high-impact research findings, yet remains a highly manual, resource-intensive process. The "AutoSynthesis: An agentic system for automated meta-analysis" (2607.15247) addresses the methodological and scalability limitations associated with traditional meta-analytic workflows. Leveraging advances in LLMs and agentic systems, AutoSynthesis operationalizes every critical stage of the meta-analytic pipeline—from literature search and eligibility evaluation to data extraction, effect size computation, synthesis, and reporting—via a multi-agent orchestration, aiming for maximized reproducibility, inspectability, and scalability.

Figure 1: Overview of the AutoSynthesis framework comprising protocol specification, literature retrieval, eligibility assessment, statistical extraction, effect size computation, synthesis, bias assessment, and PRISMA-aligned reporting; all stages are agentic and auditable.

System Architecture

AutoSynthesis implements a multi-agent framework with modular specialization for each methodological stage:

• Planning Agent: Converts natural-language research questions into structured meta-analysis protocols, including inclusion/exclusion criteria, moderators, search terms, and effect direction, supporting both unsupervised and human-in-the-loop workflows.
• Search and Retrieval Agents: Formulate keyphrase-driven queries, aggregate results from arXiv, Semantic Scholar, CrossRef, OSF Preprints, and PubMed, and ensure deduplication and filtering.
• Eligibility Agent: Executes full-text eligibility decisions per PRISMA/CONSORT criteria through LLM-powered reasoning and descriptive logging.
• Study Identification Agent: Maps empirical structures, identifying multiple independent research questions or datasets within single papers.
• Extraction/Validation/Filtering: Sequentially extract, then strictly verify, relevant statistics using a two-stage strategy (structural mapping then quantitative extraction), with robust hallucination detection, error correction, and relevance filtering.
• Analysis Agent: Computes standardized effect sizes (Cohen’s $d$, Hedges’ $g$), supports conversion from diverse statistics, and conducts REML-based random-effects models for synthesis.
• Bias Assessment Agent: Implements Egger's test, trim-and-fill, funnel/sensitivity analyses, and formal RoB (ROBINS-I, RoB2) procedures on study and meta-analytic levels.
• Report Agent: Generates PRISMA-compliant markdown reports and supporting visualizations, with exhaustive workflow logs for downstream auditing.

The architecture is implemented atop LangGraph and uses LLMs (gpt-5.4-mini, among others) for reasoning-intensive tasks but ensures all statistical computations are deterministic and code-auditable.

Empirical Evaluation

End-to-End Automation

AutoSynthesis was evaluated primarily on the synthesis of evidence concerning the persuasive impact of LLM-generated messages versus human- or non-LLM controls, using the structured question: "What is the effect of LLM-generated persuasive messages on persuasive outcomes compared with non-LLM or human-authored control messages?"

AutoSynthesis autonomously produced review protocols, formulated search queries, retrieved and screened 28 candidate studies, identified 19 eligible at full-text, and extracted 20 effect size estimates from 8 studies.

Figure 2: Workflow demonstration showing PRISMA flow diagrams, forest plot, funnel plot, cumulative/meta-analysis visualizations, moderator/heterogeneity analyses, and RoB summaries generated by AutoSynthesis.

The synthesized pooled effect was Hedges’ $g = 0.143$ (95% CI $[0.059, 0.226]$, $p < 0.001$), indicating LLM-generated persuasive messages were marginally more effective, though with substantial heterogeneity ($I^2 = 88.3\%$). Sensitivity to single-study removal was negligible, demonstrating no undue influence by any individual study.

Figure 3: Leave-one-out analysis demonstrating stability of pooled effect size ($g$ remains in $[0.13, 0.16]$) under omission of every constituent study.

Retrieval and Selection Error Analysis

Figure 4: Pipeline attrition and Venn summaries: the funnel illustrates study attrition through screening/eligibility; confusion matrix highlights overlap and discordance with manual meta-analytic inclusion.

Empirical recall for included studies, compared to a recent manual meta-analysis [Hlbling2025], was 71.4% (85.7% after qualitative corrections), and precision 62.5% (87.5% after corrections), with missed inclusions mainly attributable to unavailability of full text or eligibility featurization errors by AutoSynthesis. Additional studies unique to AutoSynthesis were primarily due to expanded search coverage and post-cutoff inclusions.

Effect Size Recovery and Agreement

Figure 5: (a) Study-level effect size agreement for matched papers: Pearson $r = 0.69$; (b) Pooled effect size with CI comparison shows strong congruence despite corpus/eligibility deviations.

AutoSynthesis’ pooled $g$ deviated by 0.12 from the expert benchmark ($g$0) but within a commonly accepted tolerance ($g$1) [Aczel2026]; confidence intervals were largely overlapping. At study-level, paired effect sizes showed $g$2 correlation ($g$3), and OLS slope $g$4. Differences at this level were dominated by eligibility divergence and document access issues, not extraction failure.

Theoretical and Practical Implications

AutoSynthesis demonstrates that agentic, LLM-augmented systems can replicate the overwhelming majority of expert-conducted meta-analytic inferences both at the study and aggregate effect size levels, provided that (a) source documents are accessible, (b) research questions are operationalizable, and (c) quantitative data are extractable using standard parsing tools.

The full automation of effect size synthesis, linkage of audit traces for each decisional step, and PRISMA-aligned, reproducible outputs fundamentally alter the attainable scalability of evidence-based knowledge synthesis. Immediate applications lie in rapid evidence aggregation for emerging topics, updating "living reviews" in response to new publications, and supporting meta-analyses in domains where methodological or language diversity impairs conventional review processes.

Architecturally, the explicit separation of reasoning, extraction, validation, and synthesis stages, with robust fail-safes and the capability for modular backbone LLM substitution (including open-weight models for full transparency [Shrestha2023]), positions AutoSynthesis as an extensible platform for the broader meta-science and scientific QA workflows.

Limitations and Caveats

Key constraints of the current framework are:

• Final performance and recall remain bottlenecked by full-text access (e.g., paywalled PDFs, non-standard layouts) and the completeness of machine-readable reporting by source studies.
• Accuracy at the eligibility and relevance-filtering stage is not immune to LLM selection bias and prompt underspecification.
• Generalization to less-structured, multi-lingual, or poorly reported outcome types may require substantive prompt/protocol engineering.
• Formal evaluation remains limited by available, high-quality, human-rated meta-analytic benchmarks spanning diverse designs/domains, and the impact of LLM training set contamination on performance is difficult to formally exclude (though audit logs enable some mitigation).
• The framework requires effective design of protocol specification and search-term generation for optimal results; expert-in-the-loop use is still valuable for edge-case design decisions.

Future Directions

Potential advances include integration of further domain-specific retrieval modules, reinforcement learning for eligibility and extraction improvement, dynamic updating capacities for "living meta-analyses", and systematic benchmarking in more adversarial domains (e.g., biomedical SRs with highly heterogeneous outcomes). Extending intermediary LLM agent logging to enable federated, human-computable override could realize true synergistic human+AI evidence synthesis.

Conclusion

AutoSynthesis represents a comprehensive, auditable, and scalable agentic solution for automated meta-analysis, achieving strong correspondence with expert benchmarks while maximizing transparency and traceability. While not obviating the role of substantive domain expertise, AutoSynthesis substantially reduces the manual burden of evidence synthesis, enabling broader, faster, and more reproducible support for evidence-based behavioral science, medicine, and policy. Continued development and benchmarking against expanded domains and review types will further clarify the attainable envelope of full-stack automated meta-analysis.

(2607.15247)