Manalyzer Framework for Automated Meta-Analysis

• Manalyzer is a multi-agent framework designed for end-to-end, automated meta-analysis by coordinating LLMs, VLMs, and specialized extraction and review agents.
• It employs hierarchical agent composition and quality assurance strategies, including hybrid review, self-proving, and feedback checking, to mitigate hallucinations and ensure data grounding.
• Empirical results show significant performance gains in screening and extraction benchmarks compared to single-LLM approaches, highlighting its scalability and reliability.

Manalyzer is a multi-agent system designed for end-to-end automated meta-analysis, integrating LLMs, vision–LLMs (VLMs), and specialized agents via tool-call orchestration to address the full spectrum of the meta-analysis pipeline. The framework addresses classical pain points in systematic review workflows such as hallucination, data grounding, and scalability. Manalyzer achieves performance beyond single-LLM baselines through hierarchical agent composition and multiple quality assurance mechanisms (Xu et al., 22 May 2025).

1. System Architecture and Agent Composition

The architecture of Manalyzer comprises specialized agents and external tools, mapped to three pipeline stages: (1) Paper Search & Screening, (2) Data Extraction, (3) Data Analysis & Reporting. Each agent executes a well-defined subtask:

Agent/Tool	Role	Technology
Keyword Generator	Topic-to-query extraction	LLM
Paper Downloader	Automated search and download	APIs (CrossRef, arXiv)
PDF Parser (MinerU OCR)	Text+figure+table segmentation	OCR
Literature Reviewer	Relevance/reliability screening (Hybrid Review)	LLM
Data Extractor	Structured extraction, grounding (Hierarchical Extraction, Self-Proving)	LLM+VLM
Checker	Accuracy/consistency/omission feedback	LLM
Data Analyst	Statistical analysis code synthesis	LLM
Reporter	Output synthesis, report composition	LLM/VLM

Agent interactions are sequential and recurrent, with feedback loops at critical validation stages. Hierarchical extraction strategies allow hybrid text/vision parsing, and an LLM-based checker implements iterative refinement.

2. Hallucination Mitigation and Quality Assurance Strategies

Manalyzer incorporates four orthogonal strategies to target hallucinations and ensure grounded extraction:

1. Hybrid Review (Screening):
   • Independent Review: Each paper $P_i$ is assigned a relevance score $s_{1,i}\in[1,10]$ and reliability score $s_{2,i}\in[1,10]$ by the LLM.
   • Knapsack-based Paragraph Selection: For long PDFs, paragraphs $p_j$ are scored by value $v_j$, with a 0/1 knapsack maximizing $\sum_j v_j x_j$ subject to token budget $B$.
   • Comparative Review: Batches of up to 20 papers are scored with inter-paper relative rankings $s_{r,i} \in [0,1]$. The final score: $s^{\text{final}}_i = s_{r,i}(s_{1,i} + s_{2,i})$. Papers are accepted if $s^{\text{final}}_i \geq \theta_\text{screen}$.
2. Hierarchical Extraction (Data Extraction):
   • All tables/images are transduced to Markdown with VLMs.
   • Stage-1 mask: Input blocks (text/table/image) are flagged relevant by LLM ($s_{1,i}\in[1,10]$0).
   • Stage-2 extraction: Only masked blocks are input to the extractor for tabulation.
3. Self-Proving: Every numeric value extracted must include provenance (specific section, table, or figure location), making each entry verifiable.
4. Feedback Checking: Checker agent rates extracted tables $s_{1,i}\in[1,10]$1 on accuracy, consistency, and completeness (each $s_{1,i}\in[1,10]$2–$s_{1,i}\in[1,10]$3), issues suggestions, and triggers refinement up to three loops. $s_{1,i}\in[1,10]$4 of cases require at least one feedback loop; most are corrected on the first iteration.

3. End-to-End Workflow

The full workflow is encapsulated in modular, agent-driven pseudocode with commentary on key computations. In outline:

• Keywords are generated for the user topic.
• Papers are found and downloaded.
• Each PDF is parsed to text, figures, and tables.
• Paragraph selection (knapsack) and batch screening (hybrid review) yield a pool of accepted studies.
• Relevant data blocks are selected and hierarchically extracted, with enforced provenance.
• The checker agent provides iterative corrections.
• All structured data are harmonized and analyzed programmatically.
• A multimodal report is generated, synthesizing findings and code execution results.

The pseudocode in the source delineates exact procedural steps and agent interactions (Xu et al., 22 May 2025).

4. Benchmark Construction and Evaluation Metrics

Manalyzer is evaluated using a comprehensive benchmark suite:

• Screening Benchmark: 182 candidate papers on “PM$s_{1,i}\in[1,10]$5 pollutant content in China, 2003–2014,” with expert labels (69 usable). Metrics include accuracy, precision, recall, F1.
• Extraction Benchmark: 729 papers across three domains:

Domain	#Papers	#Tables	#Images	#DataPoints
Atmosphere	111	331	754	1030
Agriculture	507	2452	1377	9082
Environment	111	553	461	1330
Total	729	3336	2592	11,442

Difficulty levels span extraction from text only, tables/images, and computed values (requiring unit conversion, aggregation). The extraction metric is hit rate, $s_{1,i}\in[1,10]$6.

5. Empirical Gains and Ablation Analysis

Manalyzer exhibits marked performance improvement compared to leading LLM-only baselines:

• Screening: F1 score of 76.8%, outperforming GPT-4o by +13.6pp and the best open LLM by +22.8pp.
• Extraction: Manalyzer establishes gains across all levels; for instance, 77.66% hit rate in Atmosphere Level 1 against 59.6% for GPT-4-V (Level 2: 70.65% vs. 45.0%, Level 3: 3.42% vs. 0.16%).
• Ablation: Incremental contributions are quantifiable:

Component	Level 1	Level 2	Level 3
Baseline (single LLM)	44.6	45.0	0.0
+Hierarchical Extract	65.9	55.3	0.5
+Self-Proving	71.3	67.9	1.1
+Feedback Checker	77.7	70.6	3.4

Hierarchical extraction delivers +21pp on Level 1, self-proving adds +5.4pp, and the checker yields +6.4pp. Case studies confirm hallucination mitigation, citation enforcement, and robust error correction especially in computed-value scenarios.

6. Architectural Lessons from Related Research

Best practices identified in the domain of dynamic binary instrumentation frameworks, particularly the COBAI architecture (Crăciun et al., 2023), suggest potential extensions for future iterations of Manalyzer:

• Adoption of plugin–orchestrator modularity (controller + dedicated plugins per concern) facilitates extensibility and maintenance.
• Unified, JSON-driven configuration and socket-based orchestration supports remote/batch processing.
• Distinct instrumentation-level API/hooks allow controlled intervention and facilitate transparency.
• “Shield” plugins and enclave-style memory isolation (in DBI, to resist detection) may have analogs in securing meta-analysis pipelines or enforcing provenance integrity.

A plausible implication is that some of these architecture patterns, especially those enhancing transparency, modularity, and process validation, could further strengthen the reliability, auditability, and robustness of automated meta-analysis systems.

7. Significance and Implications

Manalyzer demonstrates that end-to-end, tool-call coordinated, multi-agent frameworks can decisively advance the fidelity and scalability of meta-analysis in scientific research. The principled integration of hybrid review, hierarchical extraction, self-proving, and feedback checking yields substantial empirical gains over single-agent LLM approaches. Comprehensive benchmarking provides a foundation for rigorous, replicable system evaluation across text and multimodal sources. These findings are relevant for the future design of automated scientific reasoning systems and systematic review tools, especially where hallucination and data provenance are primary concerns (Xu et al., 22 May 2025).