M3MAD-Bench: Multi-Agent Debate Benchmark

• M3MAD-Bench is a unified benchmark framework that evaluates multi-agent debate methods across diverse domains and supports both text-only and vision-language tasks.
• It standardizes protocols and metrics—such as accuracy, token consumption, and latency—to enable controlled cross-modality comparisons of debate frameworks.
• Its findings highlight that collaborative debate strategies outperform adversarial ones, underscoring the need for robust aggregation to counteract collective delusion.

M3MAD-Bench is a unified and extensible benchmark for evaluating Multi-Agent Debate (MAD) methods across multi-domain tasks, multi-modal inputs, and multi-dimensional metrics. It was introduced to address two limitations in prior MAD research: evaluations conducted under fragmented and inconsistent settings, and a predominant focus on single-modality scenarios that rely on textual inputs only. The benchmark establishes standardized protocols over five core task domains—Knowledge, Mathematics, Medicine, Natural Sciences, and Complex Reasoning—and systematically covers both pure text and vision-language datasets, enabling controlled cross-modality comparison. It evaluates MAD methods on nine base models spanning different architectures, scales, and modality capabilities, and measures not only accuracy but also token consumption and inference time, yielding a performance–cost perspective on multi-agent reasoning systems (Li et al., 6 Jan 2026).

1. Motivation and problem setting

Multi-Agent Debate has been proposed to improve LLM reasoning via structured exchanges among multiple agents. In the formulation used by M3MAD-Bench, MAD is treated as an agent-level reasoning and coordination paradigm in which multiple agents engage in structured debate to improve answer quality and support complex reasoning (Li et al., 6 Jan 2026).

The benchmark is motivated by two specific deficiencies in earlier evaluations. First, prior studies used heterogeneous datasets, base models, hyperparameters such as agent number and temperature, and non-uniform protocols, making cross-comparison impossible. Second, almost all existing benchmarks focused on text-only tasks and ignored the increasingly important vision-language domain. M3MAD-Bench was created to fill these gaps by providing a unified, extensible evaluation framework tailored to MAD methods, with Multi-Domain coverage, Multi-Modal support, and Multi-Dimensional metrics (Li et al., 6 Jan 2026).

This positioning is methodologically significant because it shifts MAD evaluation from isolated case studies to a controlled benchmark regime. A plausible implication is that claims about the general effectiveness of debate-based reasoning can be tested under substantially more comparable conditions than in earlier work.

2. Domain coverage and dataset composition

M3MAD-Bench defines five core task domains and instantiates them with 13 datasets in total: 7 text-only and 6 vision-language. By pairing analogous tasks across modalities, such as MMLU versus MME in Knowledge and MATH versus MathVista in Mathematics, the benchmark enables controlled cross-modality studies under identical protocols (Li et al., 6 Jan 2026).

Domain	Text-only datasets	Vision-language datasets
Knowledge	MMLU; MMLU-Pro	MME
Mathematics	MATH; GSM-Hard	MathVista; MathVision
Medicine	MedMCQA; MedQA	PathVQA
Natural Sciences	GPQA	—
Complex Reasoning	—	MME-Reasoning; VisualPuzzles

The Knowledge domain includes MMLU, described as covering 57 subjects from elementary to professional levels, and MMLU-Pro, described as harder and offering more options. Its vision-language counterpart is MME, which contains 14 subtasks covering perception and cognition. The Mathematics domain combines MATH, consisting of 12,500 high-school competition problems, and GSM-Hard, characterized by larger numbers, with MathVista, which includes geometry and function graphs, and MathVision, which contains real-world competition problems with diagrams. The Medicine domain pairs MedMCQA, spanning 2,400 topics from Indian exams, and MedQA, built from USMLE scenarios, with PathVQA, which combines microscopic pathology images and clinical questions. Natural Sciences is represented by GPQA, composed of expert-verified graduate-level questions in biology, physics, and chemistry. Complex Reasoning is represented on the vision-language side by MME-Reasoning, which targets inductive, deductive, abductive, and analogical reasoning, and by VisualPuzzles, characterized as “Aha!” puzzles requiring holistic visual inference (Li et al., 6 Jan 2026).

The benchmark’s domain structure is designed to cover both canonical knowledge-intensive tasks and tasks requiring visual perception or multimodal abstraction. This suggests that M3MAD-Bench is not merely testing debate as a prompt-level heuristic, but as a coordination mechanism under heterogeneous epistemic and representational demands.

3. Standardized protocols and debate frameworks

A central contribution of M3MAD-Bench is protocol standardization. For text-only tasks, prompting uses a standard multiple-choice or open-ended question format. For vision-language tasks, the image and question are jointly encoded via an LVLM front-end. All debate frameworks use the same temperature, $0.5$, and $\text{max\_tokens} = 4096$ (Li et al., 6 Jan 2026).

The benchmark compares six inference frameworks:

Framework	Configuration	Characterization
IO (Input → Output)	Single-agent	Zero-chain prompting
CoT (Chain-of-Thought)	Single-agent	Chain reasoning
SC (Self-Consistency)	$N = 5$	Independent CoT traces with majority vote
LLM Debate	$N = 3, T = 2$	Collaborative “consensus through discussion”
Div-MAD	$N = 2, T = 3$	Adversarial “tit-for-tat”
DMAD	$N = 3, T = 3$	Collaborative with enforced diversity of thought

Aggregation is standardized as well: the final answer is selected via majority voting among agents, or via a dedicated judge prompt in the case of Div-MAD. Appendix-level configuration details further specify that DMAD uses distinct reasoning styles per agent: CoT, SBP, and PoT for text; IO, CCoT, and DDCoT for vision. SC uses $N = 5$ independent CoT chains (Li et al., 6 Jan 2026).

This design allows direct comparison among single-agent baselines, self-ensemble methods, collaborative debate, adversarial debate, and diversity-enforced debate. A plausible implication is that differences in outcome can be attributed more cleanly to debate structure rather than to uncontrolled prompt or decoding variation.

4. Metrics, verification, and experimental setup

M3MAD-Bench uses multiple-choice accuracy as its primary metric, reported as percentage correct. All answers are verified with x-Verify to ensure precise grading. Beyond correctness, the benchmark explicitly measures efficiency through token consumption and inference latency (Li et al., 6 Jan 2026).

Token consumption is tracked separately for input and output tokens per query. The benchmark defines $N$ as the total number of agents times rounds, and reports:

• $$\mathrm{InputTokens} = \sum_{i=1}^{N} t^{\mathrm{in}_i}$$
• $$\mathrm{OutputTokens} = \sum_{i=1}^{N} t^{\mathrm{out}_i}$$
• $\text{max\_tokens} = 4096$0

Inference latency is defined as:

• $\text{max\_tokens} = 4096$1

Latency is measured on the GPT-4o-mini API to minimize hardware variance (Li et al., 6 Jan 2026).

The benchmark evaluates nine base models. The text-only LLMs are LLaMA3.1-8B, InternLM3-8B, Qwen2.5-7B, and Qwen2.5-14B. The vision-language LVLMs are Qwen2.5VL-7B, LLaVA-Next-7B, InternVL3-8B, and InternVL3-14B. GPT-4o-mini is included as a closed-source multimodal model. For computational efficiency, experiments use 500 randomly sampled examples per dataset. Open-source models are run on eight NVIDIA A100-40G GPUs, while GPT-4o-mini is accessed via the OpenAI API (Li et al., 6 Jan 2026).

The emphasis on both grading precision and system cost distinguishes M3MAD-Bench from evaluations that treat debate solely as an accuracy optimization problem. In this benchmark, debate is assessed as a full inference protocol with measurable resource overhead.

5. Quantitative findings across unimodal and multimodal settings

The reported results indicate that collaborative MAD methods outperform single-agent baselines on average, but the gains are limited in magnitude and vary by modality and framework. On unimodal evaluation, averaged over Knowledge, Mathematics, Medicine, and Science, LLM Debate consistently outperforms IO, CoT, and SC by approximately 1–2 points. On LLaMA3.1-8B, the reported values are IO = 51.0%, SC = 52.6%, and LLM Debate = 53.9%. Div-MAD often underperforms substantially; for example, on the same model it achieves 38.2% versus IO’s 51.0% (Li et al., 6 Jan 2026).

On multimodal evaluation, averaged over Knowledge, Mathematics, Medicine, and Reasoning, DMAD and LLM Debate lead. For Qwen2.5VL-7B, the reported values are IO = 45.2%, SC = 44.2%, LLM Debate = 47.6%, and DMAD = 48.0%. Div-MAD remains weak at approximately 39.0% (Li et al., 6 Jan 2026).

These results support a differentiated view of MAD. The benchmark does not indicate that “debate” as a broad category uniformly improves performance. Rather, collaborative frameworks such as LLM Debate and DMAD show the strongest empirical behavior, whereas the adversarial configuration Div-MAD is repeatedly weak. This suggests that the effectiveness of MAD depends substantially on interaction topology and answer aggregation, not merely on the presence of multiple agents.

6. Efficiency, robustness, and observed failure modes

M3MAD-Bench places strong emphasis on performance–cost trade-offs. Token consumption analyses show that debate methods consume 2–4× more tokens, with most of the increase concentrated in InputTokens due to accumulated debate history. Latency measurements show that MAD methods take 2× longer on GPT-4o-mini while yielding marginal accuracy gains of approximately 1–2%. The benchmark summarizes efficiency in terms of gains per additional 1k tokens consumed, reporting approximately 0.5–1.0% higher accuracy, and concludes that latency overhead is often not justified by the small performance improvements (Li et al., 6 Jan 2026).

The robustness analysis identifies several structural properties of MAD systems. Collaborative methods, specifically LLM Debate and DMAD, outperform adversarial Div-MAD in both accuracy and stability. Heterogeneous model assignment yields no systematic boost beyond the best homogeneous setting. Increasing the number of debate rounds $\text{max\_tokens} = 4096$2 does not guarantee monotonic improvement; performance often plateaus or fluctuates. By contrast, increasing agent count $\text{max\_tokens} = 4096$3 shows consistent ensemble gains, suggesting that population scale is more consequential than discussion length (Li et al., 6 Jan 2026).

Error attribution further decomposes debate failure. The largest category is Collective Delusion at 65%, where agents reinforce mutual mistakes. Selection Failure accounts for 17%, where a correct argument appears but the final aggregator misses it. Incorrect Conformity and Context Degradation account for the remaining errors (Li et al., 6 Jan 2026).

These observations challenge a common assumption that longer or more adversarial discussion is intrinsically beneficial. In the benchmark’s results, more rounds do not reliably improve outcomes, and adversarial tit-for-tat exchange can degrade them. A plausible implication is that debate quality is bottlenecked less by conversational duration than by aggregation fidelity and susceptibility to correlated error.

7. Research implications and future directions

M3MAD-Bench is presented as a systematic, reproducible platform for assessing MAD across domains, modalities, and metrics. Its conclusions provide explicit guidance for future MAD research: prioritize collaborative over adversarial debate structures to avoid “divergent noise”; focus on agent-scale $\text{max\_tokens} = 4096$4 rather than debate length $\text{max\_tokens} = 4096$5 for reliability improvements; develop stronger aggregation or fact-verification mechanisms to mitigate collective delusion and selection failures; and extend evaluations to dynamic protocols, role adaptation, and human-in-the-loop assessments in order to capture persuasion, coherence, and emergent social behaviors (Li et al., 6 Jan 2026).

The benchmark’s broader significance lies in reframing multi-agent debate as a systems problem involving accuracy, resource expenditure, and socio-cognitive pathology. By jointly measuring correctness, token consumption, and latency, and by identifying failure types such as collective delusion and selection failure, it provides a more operational basis for comparing MAD methods than fragmented prior studies. This suggests that future progress in MAD may depend as much on coordination design and answer selection as on improving the reasoning ability of individual agents.

The accompanying implementation is released at http://github.com/liaolea/M3MAD-Bench, reinforcing the benchmark’s role as infrastructure for standardized evaluation (Li et al., 6 Jan 2026).