Internalizing Self-Consistency in Language Models: Multi-Agent Consensus Alignment (2509.15172v1)

Abstract: LLMs (LMs) are inconsistent reasoners, often generating contradictory responses to identical prompts. While inference-time methods can mitigate these inconsistencies, they fail to address the core problem: LMs struggle to reliably select reasoning pathways leading to consistent outcomes under exploratory sampling. To address this, we formalize self-consistency as an intrinsic property of well-aligned reasoning models and introduce Multi-Agent Consensus Alignment (MACA), a reinforcement learning framework that post-trains models to favor reasoning trajectories aligned with their internal consensus using majority/minority outcomes from multi-agent debate. These trajectories emerge from deliberative exchanges where agents ground reasoning in peer arguments, not just aggregation of independent attempts, creating richer consensus signals than single-round majority voting. MACA enables agents to teach themselves to be more decisive and concise, and better leverage peer insights in multi-agent settings without external supervision, driving substantial improvements across self-consistency (+27.6% on GSM8K), single-agent reasoning (+23.7% on MATH), sampling-based inference (+22.4% Pass@20 on MATH), and multi-agent ensemble decision-making (+42.7% on MathQA). These findings, coupled with strong generalization to unseen benchmarks (+16.3% on GPQA, +11.6% on CommonsenseQA), demonstrate robust self-alignment that more reliably unlocks latent reasoning potential of LLMs.

• The paper introduces MACA, a post-training framework that leverages multi-agent collaborative debate to optimize self-consistency in language models.
• It demonstrates significant improvements, reporting up to +27.6% self-consistency on GSM8K and +23.7% accuracy gains on MATH.
• MACA employs a self-supervised preference learning strategy to enhance reasoning stability and generalization across diverse tasks without external labels.

Internalizing Self-Consistency in LLMs: Multi-Agent Consensus Alignment

Introduction and Motivation

The paper introduces Multi-Agent Consensus Alignment (MACA), a post-training framework for LMs that formalizes and directly optimizes self-consistency—defined as the model’s ability to produce stable, high-quality outputs across diverse sampled reasoning paths. Unlike prior work that focuses on inference-time aggregation (e.g., majority voting, multi-agent debate) or external human preference alignment, MACA leverages collaborative debate among multiple LM clones to generate rich, self-supervised training signals. The framework is motivated by the observation that probabilistic decoding in LMs yields diverse reasoning trajectories but often fails to consistently select high-quality solutions, and that inference-time aggregation does not improve the model’s internal reasoning stability.

Formalization of Self-Consistency

Self-consistency is quantified as the modal probability $S^+_{\theta,\tau}(x)$—the probability mass assigned to the majority answer under temperature sampling. This is estimated via sampling consistency $s_t^{\theta,\tau}(x)$, the fraction of $t$ sampled trajectories that agree with the majority answer, and multi-agent debate agreement $d_M^{\theta,\tau}(x)$, the fraction of $M$ agents converging on the majority answer after deliberation. High self-consistency at elevated temperatures indicates the model’s ability to explore diverse reasoning paths while reliably converging on correct solutions.

MACA Framework and Training Objectives

MACA instantiates multi-agent debate by having $M$ clones of a base LM engage in $R$ rounds of collaborative problem-solving. Each agent generates an initial response, observes peer reasoning, and updates its answer. The final responses are partitioned into consensus-supporting ($\mathcal{G}^+$) and dissenting ($\mathcal{G}^-$) trajectories based on majority vote. This yields a post-training dataset of preference pairs, which is used to optimize four objectives:

• MV-SFT: Imitation learning on consensus-supporting traces.
• MV-GRPO: Online RL with consensus-based scalar rewards.
• MV-DPO: Direct Preference Optimization using majority/minority pairs.
• MV-KTO: Unpaired preference optimization with class-balancing.

The framework is fully self-supervised, requiring no external labels, and can be iterated for further improvement.

Figure 1: Multi-Agent Consensus Alignment framework: Multiple LM clones debate, generating majority/minority reasoning trajectories, which are partitioned for preference-based post-training.

Empirical Results

Self-Consistency and Accuracy Gains

MACA post-training yields substantial improvements in self-consistency (up to +27.6% on GSM8K), single-agent accuracy (+23.7% on MATH), sampling-based inference (+22.4% Pass@20 on MATH), and multi-agent ensemble performance (+42.7% on MathQA). Preference learning methods (MV-DPO, MV-KTO) consistently outperform scalar-reward RL and imitation learning, with MV-DPO best for larger models and MV-KTO for smaller ones.

Figure 2: Llama-3B on MATHQA: Sampling consistency curves before and after MACA post-training.

Figure 3: Llama-3B on MATHQA: Post-training self-consistency improves sampling accuracy across inference regimes.

Debate Dynamics and Generalization

Post-training increases the rate of unanimous agreement in multi-agent debate (from 13.4% to 43.4% for Qwen-2B on GSM8K), reduces non-parseable responses, and enables agents to leverage peer context for improved reasoning. Gains in self-consistency on mathematical datasets transfer to unseen domains (e.g., +11.3% on GPQA, +11.6% on CommonsenseQA), demonstrating that consensus alignment is a foundational capability for general reasoning.

Figure 4: MACA drives improvements in answer completeness and agent agreement, reallocating probability mass to consensus reasoning trajectories.

Ablation Studies

• Debate-derived supervision vs. ground truth: Unsupervised majority-vote signals are comparable to ground-truth labels for post-training, supporting scalable self-supervised alignment.
• Peer context: Conditioning on peer chains-of-thought during training improves both individual and ensemble performance.
• Multi-round debate: Iterative debate yields stronger consensus signals than single-round majority vote, with continued but diminishing returns over multiple iterations.

Implementation Considerations

MACA is implemented using QLoRA for 4-bit quantized models, enabling efficient multi-agent training and inference. The debate infrastructure supports adapter hot-swapping and dynamic resource management for scalable deployment. Training parameters are robust across learning rates, LoRA ranks, and batch sizes, with stable convergence observed in all preference learning objectives.

Figure 5: MV-GRPO training curves: Reward margins and log-probabilities separate consensus from non-consensus responses.

Figure 6: MV-DPO training curves: Preference learning increases reward margin between consensus and dissenting trajectories.

Theoretical and Practical Implications

MACA demonstrates that self-consistency is an intrinsic property of well-aligned reasoning models and that it can be robustly optimized via self-supervised, debate-derived signals. The approach internalizes consensus formation, enabling LMs to autonomously improve reasoning stability, efficiency, and accuracy without external supervision. Preference learning on debate signals teaches models to produce concise chains-of-thought and correct cases where aggregation previously led to degeneration. The framework is complementary to inference-time sampling and generalizes across domains, suggesting that consensus alignment unlocks latent capabilities beyond consistency itself.

Limitations and Future Directions

MACA requires sufficient base model competence to generate meaningful consensus signals and may amplify existing biases in model outputs. It does not directly supervise intermediate reasoning correctness. Future work could explore alternative consensus mechanisms, confidence-weighted voting, heterogeneous agent ensembles, and better utilization of minority traces. The observed generalization to difficult unseen tasks indicates that consensus alignment may be a key enabler for scalable, autonomous reasoning improvement in LMs.

Conclusion

Multi-Agent Consensus Alignment provides a principled, scalable framework for internalizing self-consistency in LLMs. By leveraging collaborative debate and preference learning, MACA achieves strong improvements in reasoning stability, accuracy, and generalization, all without external supervision. The approach advances the state of LM alignment by demonstrating that models can self-improve through internal deliberation, setting a foundation for future research in autonomous reasoning and robust ensemble methods.