Self-Signals Driven Multi-LLM Debate (SID)

• The paper introduces SID, a multi-agent debate framework that uses internal self-signals like token logits and attention to control debate dynamics.
• SID employs early exit and attention-based compression, reducing redundant computation and improving accuracy on both text-only and multimodal tasks.
• Empirical results show SID outperforms conventional methods, achieving up to a 40% token reduction and significant accuracy gains.

Searching arXiv for the specified SID paper and closely related debate/self-signal work to ground the article in current papers. Self-Signals Driven Multi-LLM Debate (SID) is a multi-agent debate framework for LLMs that uses self signals produced during generation—specifically model-level confidence and token-level semantic focus—to adaptively control whether debate should occur and what debate content should be retained. In contrast to multi-agent debate protocols centered on external debate graphs, LLM-as-a-Judge modules, or generic transcript accumulation, SID uses logits and attention as endogenous control variables: low-uncertainty responses can terminate early, while ongoing debate is compressed by selecting high-attention, disagreement-relevant content from prior turns. The framework is presented as a response to redundant computation and performance degradation in conventional MAD pipelines, with the reported objective of improving both accuracy and token efficiency across text-only and multimodal benchmarks (Chen et al., 8 Oct 2025).

1. Conceptual setting and motivation

SID is situated within the broader MAD literature, where multiple LLM agents iteratively exchange responses and refine answers. The SID paper characterizes a limitation of this literature as a predominant focus on external structures—such as debate graphs and LLM-as-a-Judge—while neglecting self signals already available inside the model’s own forward pass, including token logits and attention (Chen et al., 8 Oct 2025).

This design choice is best understood against the empirical instability of unguided debate. A controlled study of homogeneous teams of $N=10$ agents over $R=3$ rounds on GSM-Hard and MMLU-Hard found that unguided peer exchange often underperformed isolated self-correction, decomposing failure into sycophantic conformity, contextual fragility, and consensus collapse; reported values included modal adoption up to $85.5\%$, vulnerability rate up to $70.0\%$, and oracle gap up to $32.3$ percentage points, while token usage reached $2.1\text{–}3.4\times$ that of self-correction (Bertalanič et al., 29 Apr 2026). In a separate adversarial setting, LLMs also showed unstable private confidence signals, with average initial confidence of $72.9\%$ against a rational $50\%$ baseline, escalation to $83\%$ by the final round, and mutual claims of at least $75\%$ victory probability in $R=3$0 of debates (Prasad et al., 25 May 2025).

These findings do not describe SID directly, but they clarify the problem SID is trying to solve. A plausible implication is that debate quality depends not only on the existence of multiple agents, but on controlling participation and context flow so that peer interaction does not become indiscriminate prompt inflation or conformity pressure. SID operationalizes that control through self-signals rather than through external orchestration alone (Chen et al., 8 Oct 2025).

2. Formal debate setup and pipeline

The SID paper begins from a standard causal-MAD formulation. Given a query $R=3$1, tokenization $R=3$2, and an agent response $R=3$3, the model produces token logits and probabilities

$R=3$4

With $R=3$5 agents $R=3$6, the round-$R=3$7 input for agent $R=3$8 in naive MAD is

$R=3$9

SID modifies this baseline in two locations: before debate, via early-exit gating, and during debate, via attention-based compression of peer content (Chen et al., 8 Oct 2025).

The pipeline reported in the paper consists of eight stages. One agent first generates an initial response to $85.5\%$0. The system then converts the response into a confidence vector $85.5\%$1. If the uncertainty-derived confidence criterion satisfies the early-exit condition, debate is skipped. Otherwise, multiple agents generate initial answers, after which each agent is prompted to focus on disagreement in the other agents’ responses. Forward attention is extracted, high-attention tokens are retained, semantically coherent spans are reconstructed through semantic preservation, and the next-round answer is generated from the compressed context. This process repeats for a predefined number of rounds, after which final outputs are returned (Chen et al., 8 Oct 2025).

The reported default inference settings are $85.5\%$2 agents, $85.5\%$3 debate rounds, and $85.5\%$4 self-consistency samples. The paper also reports $85.5\%$5 for reasoning-oriented models, $85.5\%$6 for general-purpose models, $85.5\%$7 for MLLMs, and calibrated confidence threshold $85.5\%$8 (Chen et al., 8 Oct 2025).

3. Self-signals: confidence gating and semantic focus

The first SID control variable is model-level confidence derived from token-wise uncertainty signals. The paper uses token entropy and negative log-likelihood:

$85.5\%$9

and

$70.0\%$0

These are aggregated into sequence-level features using four strategies—average, max, first, and penultimate—yielding eight confidence variants concatenated into $70.0\%$1. The paper states that this vector is statistically significant in separating correct from incorrect answers (Chen et al., 8 Oct 2025).

SID provides two early-exit rules. The first is a vocabulary-adaptive threshold based on the observation that entropy and NLL scale with vocabulary size. The threshold is

$70.0\%$2

and the termination condition is

$70.0\%$3

The second is a calibrated confidence alternative using a lightweight classifier

$70.0\%$4

with termination when

$70.0\%$5

The paper describes the vocabulary-adaptive threshold as the default practical choice because it is training-free, while the calibrated classifier requires training on a small held-out set and performs similarly in experiments (Chen et al., 8 Oct 2025).

The second SID control variable is token-level semantic focus derived from attention maps. For agent $70.0\%$6, SID constructs an input containing the query, the agent’s own previous answer, a disagreement-oriented prompt, and the other agents’ responses. The prompt is stated explicitly: “Identify the key points where they disagree with your own reasoning. Concentrate on those disagreements and decide which line of reasoning is better.” Let $70.0\%$7 be the attention matrix at layer $70.0\%$8, head $70.0\%$9; for context token position $32.3$0 and prompt token positions $32.3$1, the semantic focus score is

$32.3$2

The top fraction $32.3$3 of context tokens is then retained:

$32.3$4

Because direct token selection can fragment syntax, SID applies semantic preservation:

$32.3$5

yielding compressed input

$32.3$6

The paper also notes that some positions are excluded because of attention sinks and special-token effects (Chen et al., 8 Oct 2025).

4. Reported empirical performance

SID is evaluated on text-only benchmarks—MMLU-Pro, Math, and GPQA—and multimodal benchmarks—ScienceQA and MMStar—using 100 sampled questions per dataset. The text-only models reported are LLaMA-3.1-Instruct-8B and GPT-OSS-20B; the multimodal models are LLaVA-v1.6-Vicuna-13B and GLM4.1V-Thinking (Chen et al., 8 Oct 2025).

For LLaMA-3.1-8B, the paper reports average accuracy of $32.3$7 for COT, $32.3$8 for IO, $32.3$9 for MAD, $2.1\text{–}3.4\times$0 for DMAD, $2.1\text{–}3.4\times$1 for SID-v, and $2.1\text{–}3.4\times$2 for SID-c. For GPT-OSS-20B, the corresponding averages are $2.1\text{–}3.4\times$3, $2.1\text{–}3.4\times$4, $2.1\text{–}3.4\times$5, $2.1\text{–}3.4\times$6, $2.1\text{–}3.4\times$7, $2.1\text{–}3.4\times$8, and $2.1\text{–}3.4\times$9, respectively. On multimodal tasks, LLaVA1.6-13B improves from MAD $72.9\%$0 to SID $72.9\%$1 on ScienceQA and from $72.9\%$2 to $72.9\%$3 on MMStar; GLM4.1V improves from MAD $72.9\%$4 to SID $72.9\%$5 on ScienceQA and from $72.9\%$6 to $72.9\%$7 on MMStar. The paper also reports up to $72.9\%$8 reduction in token usage, together with lower runtime than MAD (Chen et al., 8 Oct 2025).

An ablation on MMLU-Pro with LLaMA3.1-8B isolates the contribution of compression, early exit, and semantic preservation.

Method	Accuracy	Token Ratio
Single-round CoT	37.67	0.17
MAD	39.50	1.00
MAD + Compression	41.67	0.73
MAD + Compression + Early Exit	46.83	0.53
SID w/o Semantic Preservation	34.50	0.46
SID w/o Early Exit w/ Token-level Summary	39.50	0.68
SID	46.83	0.53

The paper interprets these results as showing that compression alone helps, early exit adds substantial benefit, semantic preservation is necessary, and replacing attention-based compression with token-level summary is worse than the full method (Chen et al., 8 Oct 2025).

Sensitivity analyses report that $72.9\%$9 works best for LLaMA3.1-8B, while best performance for top-$50\%$0 compression occurs around $50\%$1 or $50\%$2. Too small a $50\%$3 makes context incomplete, whereas too large a $50\%$4 leaves too much redundancy. Qualitative case studies further indicate that easy questions are often exited early, hard questions trigger debate, and compression can reduce more tokens on “thinking” models because those models produce more verbose reasoning (Chen et al., 8 Oct 2025).

5. Relation to adjacent debate paradigms

SID belongs to a broader family of methods that attempt to extract useful supervisory structure from disagreement, but it differs from several adjacent lines of work. Self-Debate Reinforcement Learning (SDRL) addresses a training-time mismatch between isolated RLVR and debate-time reasoning by sampling multiple candidate solutions, constructing debate contexts from those self-generated trajectories, and jointly optimizing initial and debate-conditioned responses. SDRL is explicitly framed around strengthening private critique and is reported to improve both post-debate accuracy and single-model reasoning on Qwen2.5-3B and Qwen3-4B-Base (Liu et al., 29 Jan 2026). Relative to SID, SDRL uses self-generated disagreement as training supervision, whereas SID uses self-signals to control inference-time participation and context size. This suggests complementary rather than competing design goals.

Collaborative Multi-Agent Debate (ColMAD) addresses a different failure mode: debate hacking in competitive, zero-sum protocols. It reframes debate as a non-zero-sum game, emphasizes evidence verification, self-auditing, and confidence calibration, and reports a headline improvement of about $50\%$5 over previous competitive MAD in error detection (Chen et al., 23 Oct 2025). SVR-MAD, by contrast, argues that pre-debate signals such as min token log-likelihood, perplexity, and self-reported confidence are fragile under hallucination. It therefore treats those signals as priors and updates agent reliability using posterior-style evidence based on whether an answer survives peer challenge; the paper reports token reductions up to $50\%$6 while matching or improving accuracy relative to strong MAD baselines (Jiang et al., 21 May 2026). In relation to SID, SVR-MAD can be read as a posterior-guided refinement of prior-only self-signal control.

A different but conceptually relevant direction is latent debate, which treats hidden states inside a single model as latent supporting and attacking arguments and aggregates them with a Quantitative Bipolar Argumentation Framework. Latent debate is presented as a faithful structured surrogate for model thinking on True/False tasks and as a strong baseline for hallucination detection, with high consistency with the original model’s decisions and strong links between hallucinations and latent attack patterns in middle layers (Chen et al., 1 Dec 2025). Unlike SID, it is not multi-LLM debate, but it formalizes an internal debate perspective that is compatible with the broader notion of self-signals as evidence-bearing internal variables.

6. Limitations, misconceptions, and open problems

A recurring misconception in the debate literature is that adding more peer interaction should, by itself, improve reasoning. The comparative evidence is more conditional. Structured debate can help in some domains: a three-model leader-follower protocol improved ambiguity detection for weaker models, with Mistral-7B-led debates reaching $50\%$7 success and particularly strong gains on spatial ambiguity, although debate degraded Gemma2-9B from $50\%$8 to $50\%$9 (Davila et al., 16 Jul 2025). In phishing email detection, a scripted three-agent pipeline reported that mixed-model configurations outperformed homogeneous ones, with GPT-4–LLaMA-2–GPT-4 producing the best results on four of five datasets and debate alone outperforming added CoT or role prompting (Nguyen et al., 27 Mar 2025). These results show that debate benefits are task-dependent, model-dependent, and protocol-dependent rather than universal.

A second misconception is that self-signals are automatically trustworthy as epistemic state variables. The adversarial calibration study shows that private confidence can be systematically overestimated, can escalate rather than contract under opposition, and can diverge from private scratchpad reasoning (Prasad et al., 25 May 2025). The prior-versus-posterior analysis in SVR-MAD likewise argues that pre-debate confidence-like signals are reasonable on easy questions but become much less predictive on hard problems and under hallucination (Jiang et al., 21 May 2026). From this perspective, SID’s success should not be interpreted as proving that raw self-signals are intrinsically reliable in all settings. A more defensible reading is that SID uses those signals instrumentally—primarily for selective routing and compression—and that its gains depend on the conditions under which those proxies remain informative.

The open research questions follow directly from this tension. One direction is whether self-signals should remain prior-only controls, as in SID’s early exit, or be combined with posterior evidence from actual debate outcomes, as in SVR-MAD. Another is whether self-signals are sufficient without stronger protocol structure; the negative-control study of homogeneous debate suggests that structured roles, heterogeneity, stronger orchestration, or explicit dissent mechanisms may be necessary to prevent conformity and oracle loss (Bertalanič et al., 29 Apr 2026). A third is whether inference-time control should be supplemented by debate-aware training, as in SDRL, so that agents become better at benefiting from conflicting rationales before deployment (Liu et al., 29 Jan 2026). Taken together, these lines of work indicate that SID is less a final solution than a specific design point within a broader program: using internal model signals to make multi-LLM debate selective, compressed, and less wasteful without assuming that either peer exchange or self-confidence is inherently reliable.