MAMM-Refine: Multi-Agent Model Refinement

• MAMM-Refine is a multi-agent, multi-model framework designed to iteratively refine long-form outputs by running dedicated subtasks for error detection, critique, and factual rewriting.
• It integrates strategies such as majority voting and discriminative reranking to systematically improve factual consistency in tasks like summarization and question answering.
• Empirical evaluations show that the collaborative approach yields measurable improvements in accuracy, outperforming single-agent and single-model baselines on key metrics.

MAMM-Refine (Multi-Agent Multi-Model Refinement) defines a methodological framework for improving the faithfulness of outputs in long-form generation tasks by leveraging collaborative iteration among multiple LLM agents, potentially across heterogeneous model types. Its architecture systematizes the division of generative refinement into subtasks—error detection, error critique, and factual rewriting—allocating single-model or multi-model multi-agent strategies to each based on empirical benefit. The framework’s central claim is that such multi-perspective collaboration, when tuned for subtask structure, yields measurable gains in factual consistency (faithfulness) across summarization and question answering benchmarks, surpassing single-agent and single-model approaches (Wan et al., 19 Mar 2025).

1. Motivation and Problem Definition

LLMs reliably produce fluent text but are known to hallucinate, generating assertions not supported by input context—a serious liability in summarization and long-form QA. Existing “self-refinement” approaches, which iteratively revise a generation using a single LLM, have shown limited error detection capacity because they lack external perspectives and diversity of critique. While multi-agent debate has improved closed-set reasoning accuracy, its impact on open-ended generative tasks remained underexplored.

MAMM-Refine was proposed to close this gap by orchestrating multi-agent, multi-model collaboration at the level of discrete subtasks within the generative refinement process, systematically evaluating which configuration benefits each phase of refinement most for factual accuracy (Wan et al., 19 Mar 2025).

2. Formalization: Agent and Task Structure

The framework introduces two explicit sets: agents and models.

• Agent Set $A$: Individual LLM instances, each denoted $a_j$.
• Model Set $M$: Distinct LLM types (e.g., GPT-4o, Claude-3.5 Sonnet).

Given input context $X$ (document or QA prompt) and initial output $Y = (y^0,\dots,y^N)$ split into sentences, the pipeline performs three principal subtasks:

1. Detect: Each agent labels each sentence as “faithful” or “unfaithful” and provides a natural language reasoning chain for its verdict.
2. Critique: Agents generate natural-language critiques specifying the “error span” and proposing corrections for sentences flagged as unfaithful.
3. Refine: Taking critiques as input, agents rewrite flagged sentences to align with factual context.

For aggregation, the framework adopts either majority voting or score averaging for detection, and a discriminative reranking protocol for critique and refinement. In reranking, multiple candidate outputs are ranked by agent consensus, not just generated in isolation. This division is codified in functional notation—for detection, $s_{\text{detect}}(X,y^i;a_j)$, and similarly for critique and refinement with reranking.

3. Core Algorithm: The MAMM-Refine Pipeline

The canonical pipeline is as follows (Wan et al., 19 Mar 2025):

Input: Context X, generated sentences Y = (y^0,...,y^N) Agents: A = {GPT-4o, Claude} MaxRounds = 10 for each sentence y^i in Y: for agent a in A: d^i_a, r^i_a = agent labels (faithful/unfaithful), reasoning chain d^i = majority_a(d^i_a) S = set of i where d^i == 'unfaithful' for i in S: Generate K critiques c^i_k using, e.g., 2 instances of Claude Rank critiques via agent consensus: c^i* = argmax sum_a(score_a(c^i_k)) for i in S: Generate K candidate rewrites using GPT-4o guided by c^i* Rank rewrites via agent consensus: y^i_r* = argmax sum_a(score_a(y^i_rk)) Output: Y_r (refined output)

Key architectural insight: multi-agent, multi-model (MAMM) setup is optimal for error detection, while single-model multi-agent reranking (MASM) improves critique and refinement, likely due to increased direct contrast in ranking tasks.

4. Experimental Methodology and Evaluation Metrics

The framework was evaluated using both intrinsic (subtask-level) and extrinsic (end-to-end) metrics.

• Agents: GPT-4o and Claude-3.5 Sonnet; sometimes Gemini-1.5 or Llama3.1-8B for ablations.
• Datasets: TofuEval for intrinsic subtasks (sentence-level faithfulness labeling and error span critique), MediaSum and MeetingBank for summarization, UltraChat (multi-turn dialogues), and ELI5/WebGPT for LFQA.
• Metrics:
  • Detect: Balanced Accuracy (BACC) vs. human-labels.
  • Critique: Error-Match (EM%), Error-Mismatch (EMM%), No-Error false positives.
  • Refine: MiniCheck faithfulness (automated), GPT-4 Likert scale, and VeriScore.
  • Reranking: Acc@1 (fraction of sets where the correct output ranked top by agents).
  • Human Reference: Intraclass agreement ≈ 0.80 for error spans; Kendall τ ≈ 0.46, correlating human-LLM faithfulness rankings.

The pipeline optimizes for faithfulness exclusively, not fluency or style (Wan et al., 19 Mar 2025).

5. Quantitative Results and Empirical Findings

MAMM-Refine demonstrated statistically significant improvement over single-agent and single-model baselines:

• Detect: Multi-agent multi-model (MAMM, e.g., GPT-4o + Claude) outperformed single-agent and single-model multi-agent (SA, MASM); e.g., BACC 80.2 (MAMM) vs. 76.5–77.7 (SA) on MeetingBank.
• Critique: Best intrinsic error-detection achieved by MASM with reranking (2×Claude: EM% 99.2).
• Refine: Intrinsic improvements in MiniCheck/GPT-4 Likert scores; MAMM-Refine yielded 82.4/4.4 on MediaSum (statistically significant), outperforming all single-agent and single-model variants.

Ablations established that agent strength parity is essential—heterogeneous (weak+strong) agent ensembles degrade performance. Introducing a third agent gave marginal improvement (e.g., MiniCheck 84.9→85.2).

6. Qualitative Analysis and Case Studies

Representative cases illustrate pipeline operation:

1. Summarization Correction: For a misdated event (“March 12” instead of “March 10”), MAMM-Refine detects the factual error, localizes the span, and generates an accurate rewrite.
2. Fact Correction in Financial Summaries: Corrects monetary values (“$15M” to “$5M”) directly from input context, mirroring human-annotated error spans.
3. QA Filtering: Faithful answers are correctly passed through with no alteration, indicating low false positive rate in detection.

Critique quality matches or surpasses human-label error highlighting, with agents providing both localization (span) and constructive corrections.

7. Limitations, Generalization, and Future Directions

The approach is computationally intensive due to multiple LLM query rounds (typically converges in 2-3), and its benefits are limited to factual faithfulness—not broader stylistic or discourse-level goals. The dependence on LLM-based faithfulness metrics, while correlated with human judgment, is not a perfect proxy.

MAMM-Refine generalizes across multiple long-form generation domains (dialogue, summarization, QA), but gains are most robust when agent pools contain models of comparable strength. Future work aims to improve pipeline efficiency, extend to additional text properties (beyond faithfulness), incorporate retrieval/tool-augmented agents, and develop heuristics for optimal debate round termination (Wan et al., 19 Mar 2025).

A plausible implication is that similar multi-agent coordination heuristics could be beneficial in other open-ended LLM-based generation tasks, particularly where model uncertainty or hallucination risk is high, but these would require task-specific empirical validation.