ConfMAD: Confidence-Aware Multi-Agent Debate

• ConfMAD is a multi-agent debate framework that extends traditional methods by integrating calibrated confidence scores as a first-class signal during each turn.
• It employs mechanisms like LN and self-verbalized confidence, using calibration techniques such as Platt scaling to align confidence with empirical accuracy.
• Experimental results reveal that calibrated confidence improves overall debate performance, boosting accuracy, consensus, and error correction rates across multiple benchmarks.

ConfMAD is a confidence-aware multi-agent debate framework for generative LLMs that extends Multi-Agent Debate (MAD) by requiring each agent to express an explicit confidence score alongside its reasoning trace and answer at every turn (Lin et al., 17 Sep 2025). It is motivated by two paired observations: some LLMs may have superior task-specific knowledge or reasoning capability yet fail to communicate that advantage during debate, and inappropriate confidence expression can instead induce either stubborn persistence in incorrect beliefs or premature convergence on suboptimal answers, thereby reducing debate effectiveness and overall system performance (Lin et al., 17 Sep 2025). Within this formulation, confidence is not an auxiliary annotation but a first-class signal in deliberation, calibration, and final answer selection.

1. Problem setting and relation to traditional MAD

Traditional MAD, as described in the ConfMAD presentation of prior work by Du et al. (2023), consists of $n$ LLM agents $M_1,\dots,M_n$ that independently generate initial answers and then debate sequentially by reading a shared history and appending new arguments (Lin et al., 17 Sep 2025). After a fixed number of rounds $T$, a final answer is chosen, for example by majority vote.

ConfMAD preserves the multi-agent deliberative structure but alters the information exchanged during debate. At every turn, each agent emits a reasoning trace $R_i^r$, an answer $A_i^r$, and a confidence score $C_i^r$ (Lin et al., 17 Sep 2025). The framework further assumes that raw confidence is not necessarily reliable and therefore subjects it to explicit calibration before it is exposed to other agents or used in downstream decision-making.

The central design premise is that debate quality depends not only on the content of arguments but also on whether stronger agents can communicate epistemic certainty in a usable form. This suggests that ConfMAD is best understood as a debate-control mechanism rather than merely a confidence-reporting add-on: confidence affects how agents interpret one another’s claims, how disagreement evolves over rounds, and which terminal answer is selected.

2. Debate protocol and system architecture

ConfMAD integrates confidence expression throughout the debate process (Lin et al., 17 Sep 2025). In the one-by-one debate workflow, the initial round $r=0$ is defined agentwise as

$D_i^0 = \bigl(R_i^0, A_i^0, C_i^0\bigr),$

where

$C_i^0 = \mathrm{Calibrate}(C_i^{\prime 0}).$

The initial history is then

$H_0 = [D_1^0,\dots,D_n^0].$

For rounds $M_1,\dots,M_n$0, each agent acts in turn:

$M_1,\dots,M_n$1

with

$M_1,\dots,M_n$2

and the shared history is updated by appending the new debate item:

$M_1,\dots,M_n$3

The final decision rule departs from majority-vote MAD. ConfMAD selects the answer of the agent with the highest calibrated confidence at the terminal round:

$M_1,\dots,M_n$4

The returned prediction is $M_1,\dots,M_n$5 (Lin et al., 17 Sep 2025). When confidences tie, the answer is chosen uniformly at random.

Agents are explicitly instructed through the system prompt to “PAY SPECIAL ATTENTION to these confidence scores…,” so confidence is intended to influence subsequent arguments rather than merely serve as metadata (Lin et al., 17 Sep 2025). An alternate broadcast debate mode, in which all agents speak in parallel each round, is also supported, although the primary comparisons use the one-by-one setting.

This architecture makes calibrated confidence operational at three levels: local utterance production, inter-agent influence, and final aggregation. A plausible implication is that ConfMAD changes both the semantics and the control flow of debate, because an agent’s contribution is evaluated through jointly visible content and calibrated certainty.

3. Confidence quantification and calibration

ConfMAD supports two confidence quantification mechanisms and three calibration methods (Lin et al., 17 Sep 2025).

The first mechanism is key length-normalized sequence probability, denoted here as LN confidence. If the answer tokens are $M_1,\dots,M_n$6, the raw sequence probability is

$M_1,\dots,M_n$7

The raw confidence is then defined as

$M_1,\dots,M_n$8

after which it may optionally be multiplied or scaled into $M_1,\dots,M_n$9 before calibration (Lin et al., 17 Sep 2025). This definition uses token-level model probabilities and normalizes for answer length.

The second mechanism is self-verbalized confidence, denoted in the paper as SV confidence. Under this scheme, each LLM is prompted to append the string “Confidence score: $T$0.” The raw output is therefore a scalar $T$1 (Lin et al., 17 Sep 2025). In contrast to LN confidence, SV confidence is explicitly generated by the model as part of the textual response.

Because raw confidence may be poorly aligned with empirical correctness, ConfMAD applies post-hoc calibration. Three calibration methods are specified.

Platt scaling is given by

$T$2

where $T$3 and $T$4 are fitted on a held-out validation set (Lin et al., 17 Sep 2025).

Histogram binning partitions raw confidences into $T$5 intervals and replaces each raw score by the empirical accuracy of its bin (Lin et al., 17 Sep 2025).

Temperature scaling, used only for LN confidence, rescales answer-token logits $T$6 according to

$T$7

and produces calibrated confidence from $T$8 (Lin et al., 17 Sep 2025).

These design choices separate confidence extraction from confidence correction. This suggests that ConfMAD does not assume self-reported or probability-derived confidence is intrinsically trustworthy; rather, confidence becomes useful once statistically aligned with observed accuracy.

4. Algorithmic realization

The high-level algorithm consists of an initialization phase, an iterative debate phase, and a confidence-based terminal selection step (Lin et al., 17 Sep 2025). The inputs are models $T$9, question $R_i^r$0, prompt $R_i^r$1, and maximum rounds $R_i^r$2, and the output is a final answer $R_i^r$3.

In the initial round, each agent is queried independently to produce $R_i^r$4, after which the raw confidence is calibrated to obtain $R_i^r$5, and the tuple $R_i^r$6 is stored (Lin et al., 17 Sep 2025). The initial debate history is the concatenation of all initial tuples.

In each debate round $R_i^r$7, the current history is copied into a working history $R_i^r$8. Then, for each agent in sequence, the model is invoked conditioned on the question, prompt, and working history to generate $R_i^r$9. The confidence is calibrated, the tuple $A_i^r$0 is formed, and $A_i^r$1 is extended by concatenation with $A_i^r$2 (Lin et al., 17 Sep 2025). At the end of the round, $A_i^r$3 is set to the accumulated working history.

Most experiments use two one-by-one debate rounds plus the initial round, so the principal setting is shallow rather than long-horizon deliberation (Lin et al., 17 Sep 2025). The remaining hyperparameters are the calibration parameters $A_i^r$4 for Platt scaling, the bin counts for histogram binning, or the temperature parameter $A_i^r$5 for temperature scaling.

The algorithm therefore instantiates a recurrent debate process in which calibrated confidence is recomputed at every turn. A plausible implication is that confidence is not treated as a static prior over agents but as a trajectory-dependent variable that may change as agents absorb others’ arguments.

5. Experimental configuration and empirical results

The reported evaluation covers four benchmarks: BIGGSM with 400 test and 200 validation examples, BBH with 2,000 test and 1,000 validation examples, MMLU with 2,000 test and 1,000 validation examples, and MATH with 1,000 test and 1,000 validation examples (Lin et al., 17 Sep 2025). Validation splits are used to train the calibration models.

Two principal model pairs are reported: GPT-4o-mini + LLaMA-3.1-70B, and GPT-4o-mini + Phi-4 (Lin et al., 17 Sep 2025). The baselines are Chain-of-Thought (CoT), No Confidence Debate (No Conf), Interventions (Inter), ChatEval (CE), and Multi-Persona (MP) (Lin et al., 17 Sep 2025).

The evaluation metrics are accuracy of the final answer, consensus rate, correction cases, Win Rate (WR), and calibration quality measured by Expected Calibration Error (ECE) (Lin et al., 17 Sep 2025). ECE is defined as

$A_i^r$6

The key quantitative findings reported in Tables 1–3 are that ConfMAD variants, such as LN+Platt and SV+Platt, outperform all baselines on most datasets (Lin et al., 17 Sep 2025). On MMLU with GPT-4o-mini+LLaMA, LN+Platt achieves 0.833 versus No Conf at 0.783, a gain of 5 points. On BBH, LN+Platt reaches 0.763 versus No Conf at 0.730, a gain of 3.3 points. Similar gains of 2–5 points are reported on BIGGSM and MATH.

Beyond end-task accuracy, debate-level indicators also improve. Consensus improves by up to +11%, exemplified by MMLU in Figure 1, and correction cases increase by up to +20%, exemplified by MMLU with Phi in Figure 2 (Lin et al., 17 Sep 2025). Because correction cases measure how often a wrong initial answer is flipped to correct by debate, these results indicate that confidence does not merely stabilize agreement; it can also increase the probability that deliberation repairs initial error.

6. Analytical findings, debate dynamics, and interpretive issues

The calibration ablation in Table 6 reports that Platt scaling is the most robust calibration method, temperature scaling is often second best, and histogram binning is unstable (Lin et al., 17 Sep 2025). The paper also notes that in some settings uncalibrated (“Vanilla”) confidences perform nearly as well in final accuracy, but exhibit poor WR. This is analytically important because it distinguishes raw accuracy from decision reliability under disagreement.

Win Rate is defined as the fraction of disagreements in which the correct agent had higher confidence (Lin et al., 17 Sep 2025). The reported analysis shows that higher WR correlates with higher final accuracy, and calibration improves WR substantially; on MMLU, raw LN confidence yields WR of approximately 0.44, whereas LN+Platt yields approximately 0.62 (Lin et al., 17 Sep 2025). This suggests that confidence-aware debate is beneficial not simply when agents become more confident, but when the confidence ordering between disagreeing agents better matches correctness.

The calibration-quality analysis in Appendix E reports that Platt scaling reduces ECE from approximately 20–50% down to single-digit percent on validation and test (Lin et al., 17 Sep 2025). Since ECE measures the gap between empirical accuracy and predicted confidence, this finding indicates that the confidence values used inside the debate become materially better aligned with observed performance after calibration.

The paper’s analysis of debate dynamics reports that accuracy peaks after 2–3 rounds, that ConfMAD still improves over No Conf with three agents (4o-mini, LLaMA, Phi), and that very coarse confidence granularity, such as 0–10, can sometimes destabilize performance (Lin et al., 17 Sep 2025). These observations constrain interpretation. First, longer debate is not necessarily better; beyond a small number of rounds, additional exchange may not improve accuracy. Second, the approach is not restricted to two-agent settings. Third, the informativeness of the confidence channel depends not only on calibration but also on representational granularity.

Case studies in Appendix G, Figures 8–9, describe examples in which one agent observes another’s high calibrated confidence and re-evaluates its reasoning, leading to correct consensus (Lin et al., 17 Sep 2025). This provides a mechanistic illustration of how confidence can alter argumentative uptake rather than merely post-process outputs.

A common misconception would be to equate confidence expression with beneficial assertiveness. The underlying results do not support that simplification. The motivating abstract explicitly states that inappropriate confidence expression can make agents stubbornly maintain incorrect beliefs or converge prematurely on suboptimal answers (Lin et al., 17 Sep 2025). ConfMAD addresses this not by maximizing confidence, but by calibrating and exposing confidence so that it becomes a more reliable signal inside deliberation. Overall, the reported evidence supports the conclusion that explicit confidence expression together with simple post-hoc calibration can enhance both individual agent performance and the quality and reliability of multi-agent debates (Lin et al., 17 Sep 2025).