Multi-Agent Debate Consistency (MADC)

• MADC is defined as the property by which a multi-agent debate system yields correct and stable outcomes despite perturbations in agent responses and debate structure.
• Advanced mechanisms such as Free-MAD and Path Consistency Ordering employ score-based updates, dynamic role allocation, and retrieval augmentation to enhance inter-agent agreement.
• Empirical evaluations demonstrate that MADC frameworks can improve consensus accuracy by up to 22%, while challenges like computational overhead and lexical divergence persist.

Multi-Agent Debate Consistency (MADC) denotes the property by which a multi-agent debate system—typically composed of LLM agents—produces final decisions that are both correct and stable under perturbations to agent responses and debate structure. MADC is essential for robust, high-fidelity reasoning in domains where correctness and reproducibility of group decisions are paramount, such as mathematical problem-solving, fact verification, and complex long-form question answering. Central to MADC is the quantification and enforcement of agent agreement, tracking the evolution of reasoning trajectories, and designing mechanisms that mitigate conformity-driven error propagation.

1. Formal Definitions and Core Metrics

MADC is characterized by both inter-agent consistency (agreement among agents at convergence) and trajectory consistency (monotonic progression of each agent’s reasoning toward the correct answer, rather than oscillations or silent convergence on erroneous consensus) (Cui et al., 14 Sep 2025). In several frameworks, consistency is formalized as a score over discrete answer choices or reasoning paths.

Let $\{A_i\}_{i=1}^N$ be $N$ LLM agents, each producing a sequence of viewpoints $\{V_{i,j}\}_{j=1}^{m}$ over $m$ rounds. For agent $i$ and round $j$, local path consistency is:

$$c_i^{(j)} = \frac{1}{N-1} \sum_{k \ne i} \mathbf{1}(V_{i,j} = V_{k,j})$$

Aggregated global consistency scores can be computed as:

$C_i = \sum_{j=1}^{m-1} c_i^{(j)}$

In the context of factuality evaluation for long-form claims, MADC can be formulated as the average agreement over $T$ atomic claims among $N$ evaluating agents (Ning et al., 27 Oct 2025):

$$\mathrm{MADC}(a) = \frac{1}{T} \sum_{j=1}^T \frac{1}{\binom{N}{2}} \sum_{1 \le n < m \le N} \mathbf{1}[p^n_j = p^m_j]$$

where $p^n_j$ is the judgment of agent $n$ on claim $j$.

Consistency Score $C$ (MADKE) (Wang et al., 2023):

$$C = \frac{1}{N} \sum_{k=1}^N \mathbf{1}(A_1^{(k)} = A_2^{(k)} = \dots = A_m^{(k)})$$

2. Limitations of Consensus-Based MAD Protocols

Conventional multi-agent debate systems operate in two phases: multi-round debate, followed by majority voting on the final-round responses. Agent outputs in round $k$ are sampled from conformity-augmented distributions:

$$P_{a_i}(r | C^{(k-1)}, p) = \frac{1}{Z} P_{\text{in}}(r | q, p) \exp\left[\beta(p) S_{\text{con}}(r, C^{(k-1)})\right]$$

Here, $P_{\text{in}}$ encodes the agent’s intrinsic reasoning, $S_{\text{con}}$ reflects alignment to peer outputs, and $\beta(p)>0$ enforces conformity. Majority voting ignores signals from earlier rounds, allows “silent agreement” on incorrect answers, and is vulnerable to unfairness and randomness in tie scenarios (Cui et al., 14 Sep 2025). This design amplifies error propagation and can suppress the influence of the initially correct agent, leading to degraded reasoning performance.

3. Score-Based and Consistency-Aware Mechanisms

Addressing these limitations, advanced frameworks replace consensus-based protocols with consistency-aware mechanisms:

Free-MAD Score-Based Decision (Cui et al., 14 Sep 2025)

Free-MAD eliminates consensus dependency by maintaining a full answer matrix $A \in \mathbb{R}^{N \times (R+1)}$ and a score dictionary $S$ over all candidate answers. At each round $k$:

• If $k=0$, increment $S[\hat r]$ by $w_1 f(k)$.
• If the agent switches answers ($\hat r \ne r_p$), penalize old $S[r_p]$ and reward new $S[\hat r]$ with weight functions.
• If maintaining answer, reward $S[\hat r]$.

$f(k) = (k+1)^{-1}$ downweights later rounds, controlling conformity influence. The final decision selects the candidate with maximal cumulative score after all rounds.

Path Consistency Ordering (Zhang et al., 14 Nov 2025)

MADC introduces dynamic role allocation, where agents are permuted per round based on accumulated path consistency scores $S_i$. This simulates “Truth Last” strategies—putting the most consistent agent last—without prior knowledge of the ground truth. The system aggregates local consistency over intermediate rounds and reorders agents to concentrate high-consistency speakers near decision points.

4. Retrieval-Augmented and Knowledge-Gated Consistency

Retrieval augmentation directly impacts consistency by evidentiary alignment. In MADKE, a shared evidence pool (composed of Wikipedia DPR and Google API retrievals) is distributed to agents, enabling them to fact-check against common ground (Wang et al., 2023). Adaptive knowledge selection enables agents to gate which evidences are incorporated, maximizing factual convergence and mitigating cognitive island effects. Experimental results show this approach increases agent agreement rates $C$ by up to +10.2% absolute (FEVER) and +8.8% (FEVEROUS). Real-time retrieval can further improve consistency but introduces noise sensitivity.

5. Weighted Consistency and Factuality Metrics in Long-Form Verification

In MAD-Fact, multi-agent debate consistency is foundational to factual assessment for long-form texts (Ning et al., 27 Oct 2025). Responses are decomposed into atomic claims, evaluated by diverse role-playing agents, and aggregated via majority or weighted voting. A fact importance hierarchy (pyramid model) weights claims by their consensus across multiple expert references:

$\omega_\ell = G - \ell + 1 \quad \ell=1,\dots,G$

Weighted precision and recall metrics quantify factual agreement, and overall consistency is measured as the average pairwise agent agreement across all claims. Empirical results demonstrate MAD-Fact’s consistency mechanism achieves superior F1 scores on atomic-claim datasets and excels in cross-domain, multi-lingual settings. Ablations indicate that knowledge retrieval, role diversity, and structured debate all positively contribute to consistency.

6. Theoretical and Empirical Insights, Limitations, and Future Directions

Theoretical analysis shows that eliminating consensus reliance (setting $\beta \leq 0$), trajectory-aware scoring, and dynamic, consistency-driven role allocation collectively reduce error amplification and increase robustness under adversarial attack or communication failure (Cui et al., 14 Sep 2025, Zhang et al., 14 Nov 2025). MADC mechanisms generalize across debate topologies (dense, sparse, chained), agent pools, and evidence-gated frameworks. Limitations include sensitivity to output format mismatches (lexical divergence), computational overhead for large agent pools ($O(n^2)$ in some score aggregations), and possible collective “agreement” on incorrect answers.

Extensions under investigation include embedding-based semantic consistency, adaptive weighting via meta-learning, robust aggregation through spectral or topological methods, and dynamic agent composition. Open challenges remain regarding semantic generalization, adversarial robustness, domain adaptation, and multilingual consistency.

7. Summary Table: MADC Mechanisms and Performance Gains

Framework	Core Mechanism	Maximum Consistency Gain	Key Limitation
Free-MAD (Cui et al., 14 Sep 2025)	Score-based, anti-conformity	+16% accuracy (R=1, anti-conformity)	“Stubbornness” under pure anti-conformity
MADC (Zhang et al., 14 Nov 2025)	Path consistency ordering	+9.6% vs. Fixed; up to +22% (Truth Last)	Output format dependency, scalability
MADKE (Wang et al., 2023)	Shared evidence, self-selection	+10% (FEVER), +8.8% (FEVEROUS)	Noise sensitivity, retrieval quality
MAD-Fact (Ning et al., 27 Oct 2025)	Weighted claim aggregation	Best in 8/10 F1 comparisons	Lexical divergence, debate cost

All mechanisms exhibit improved stability, fairness, and scaling for multi-agent reasoning and verification tasks. MADC provides a rigorous, extensible foundation for robust group decision-making in model-ensembled LLM environments.