Anti-Consensus Preference Optimization

• ACPO is a family of preference optimization methodologies designed to counteract failure modes such as confabulation consensus and likelihood displacement.
• It introduces dynamic, asymmetric constraints that stabilize model alignment, preserving visual anchors and accurate minority responses in multi-agent and vision-language systems.
• Empirical results show improved benchmarks and robustness over DPO, validating ACPO's efficacy in addressing consensus-trap scenarios.

Anti-Consensus Preference Optimization (ACPO) encompasses a family of preference optimization methodologies designed to counteract prevalent failure modes in model alignment—most notably, confabulation consensus in multi-agent LLM systems and likelihood displacement in vision-language alignment. Across application domains, ACPO departs from symmetric preference objectives by introducing dynamic, asymmetric constraints, with empirically validated increases in robustness and accuracy under regimes where standard methods such as Direct Preference Optimization (DPO) exhibit pathological behaviors.

1. Conceptual Foundations and Motivation

Standard preference optimization techniques, especially DPO, have achieved broad adoption for aligning large language and vision-LLMs with human and automated feedback. DPO operates by maximizing the likelihood margin between preferred (chosen) and dispreferred (rejected) outputs, typically utilizing a symmetrically structured gradient based on paired response log-likelihoods. However, this symmetry produces severe drawbacks under two distinct, yet structurally analogous, phenomena:

• Likelihood Displacement and Visual Anchor Collapse: In multimodal models (especially vision-language), optimization under DPO tends to decrease not just the likelihood of the rejected sample, but also the chosen one, particularly when the two share visual tokens. This erodes visual grounding, leading to what is termed "Visual Anchor Collapse," manifesting as an abandonment of visual evidence in favor of generic language priors and pronounced hallucination rates (Huang et al., 23 Mar 2026).
• Confabulation Consensus in Multi-Agent LLMs: In multi-agent systems, aggregation via majority vote or naive auditor models is fundamentally susceptible to "confabulation consensus": correlated model biases or prompt effects lead multiple agents to converge on erroneous answers, which then become preferentially selected merely due to frequency. DPO and RLHF approaches trained on random preferences do not specifically target, and thus fail to remediate, this confounding effect (Yang et al., 10 Feb 2026).

These pathologies motivate the need for asymmetric, context-sensitive optimization mechanisms that spotlight evidence-based minority outputs and explicitly protect target structures (visual anchors or correct minority traces) from collateral suppression.

2. Mathematical Formulation in Multimodal Alignment

In the context of vision-LLM alignment, ACPO introduces an asymmetric constraint into the standard preference optimization objective, anchored by dynamically calibrated, complexity-aware scaling.

Let $r(y; x)$ denote the implicit DPO reward for output $y$ given context $x$: $$r(y;x)=\beta\sum_{t=1}^{|y|}\log\frac{\pi_\theta(y_t\mid x, y_{<t})}{\pi_{\rm ref}(y_t\mid x, y_{<t})}.$$ To counteract length dependence, ACPO defines a per-token target margin $\delta$ and computes the length-adaptive advantage target: $\tau(y_w, y_l) = \delta \bigl(|y_w| + |y_l|\bigr).$

A batch-wise calibration coefficient $\hat\alpha$ is then determined: $$\hat\alpha = \operatorname{clamp}\Bigl(\,\mathrm{sg}\bigl[\tfrac{r(y_w)-\tau}{r(y_l)+\epsilon}\bigr],\,0,1\Bigr),$$ where $\mathrm{sg}[\cdot]$ denotes stop-gradient and $\epsilon$ prevents division by zero.

The ACPO loss becomes

$y$0

with gradient

$y$1

This layout ensures that, once the required margin is attained, $y$2 and suppression of the rejected response ceases, preserving the chosen likelihood as a stable anchor (Huang et al., 23 Mar 2026).

3. ACPO in Multi-Agent Adjudication

In multi-agent LLM ensembles, ACPO is operationalized as a direct preference optimization over "consensus-trap" cases—those examples where the majority is wrong, but a minority branch supports the correct answer.

Given a preference triplet $y$3, where $y$4 forms a divergence packet at the first point of disagreement, $y$5 is the minority-correct branch, and $y$6 is the majority-wrong branch, the ACPO loss is defined as: $y$7 with

$y$8

Unlike traditional DPO, the dataset for ACPO is carefully mined to include only those cases where majority-vote fails and a correct minority exists, thus directly targeting the pathological failure regime (Yang et al., 10 Feb 2026).

4. Empirical Results and Evaluation

4.1 Multimodal Alignment

ACPO was evaluated on InternVL3-Instruct models (14B, 8B) across diverse preference data (≈ 320K multimodal pairs). Key hallucination and capability benchmarks demonstrate consistent improvements over DPO and other baselines:

Benchmark	DPO (14B)	ACPO (14B)	Gain
HallusionBench	69.7	70.0	+0.3
MM-IFEval	0.500	0.570	+0.07
POPE	86.89	89.22	+2.33
AMBER	89.78	90.79	+1.01

Further, training dynamics indicate that ACPO stabilizes and preserves chosen-response reward, whereas DPO exhibits steady reward degradation and margin collapse (Huang et al., 23 Mar 2026).

4.2 Multi-Agent Systems

Experiments on LLM-based multi-agent debate protocols (e.g., LLM-Debate, Group-Debate, DyLan, GPTSwarm) on GSM8K and AMC indicate that ACPO consistently outperforms DPO by 1.2–2.7 absolute points, exceeding the typical variance range observed over multiple seeds. These effects validate the targeted correction of confabulation and sycophancy biases (Yang et al., 10 Feb 2026).

Protocol	DPO Accuracy	ACPO Accuracy	Gain
LLM-Debate (GSM8K)	86.05	87.43	+1.38
LLM-Debate (AMC)	22.83	24.65	+1.82
Group-Debate (GSM8K)	86.72	89.15	+2.43
Group-Debate (AMC)	23.55	25.31	+1.76

5. Algorithmic Procedures and Implementation

5.1 Pseudocode for Vision-Language Alignment

For each batch, ACPO entails computing sequence rewards for both chosen and rejected responses, inferring a length-adaptive margin, calculating the raw and clamped scaling coefficient $y$9 (with gradient blocking), and finally applying the asymmetric loss per sample, as detailed in the pseudocode provided in (Huang et al., 23 Mar 2026).

5.2 Pseudocode for Auditor Fine-Tuning

In multi-agent auditing, ACPO proceeds by extracting all majority-failure instances ("consensus-traps"), localizing the first point of disagreement, and assembling preference triplets. The auditor model is then fine-tuned via the DPO-style loss strictly on this curated set, as outlined in the article pseudocode (Yang et al., 10 Feb 2026).

6. Practical Implications, Limitations, and Applications

ACPO is computationally efficient, introducing only lightweight per-batch scalar calculations and negligible memory cost compared to DPO. The principal hyperparameter, the per-token target margin $x$0, is robust across tasks but can be tuned to adjust target separation.

The method is modality-agnostic and applicable beyond vision-language or text settings, whenever preferences exhibit shared substructure. In multi-turn dialogue or online RL settings, ACPO suggests replacing static margin constraints with per-turn advantage targets.

Key limitations include:

• Data Dependence: Success of ACPO in multi-agent settings requires sufficient ground-truth for filtering incorrect-majority instances. Absence of diverse agent traces limits utility.
• Segmentation Accuracy: In auditor scenarios, localization of the correct divergence point is critical; algorithmic errors in tree construction can dilute effect.
• No Solution Synthesis: ACPO does not generate correct solutions but selects among model-generated candidate branches; if no correct branch exists, the method cannot recover.

Planned extensions include adaptation to policy-gradient methods (PPO), richer multi-objective preference structures, and adversarial co-training of agents and auditors.

7. Theoretical Significance and Outlook

ACPO fundamentally advances preference optimization by dynamically halting the pressure to suppress rejected targets when a sufficient gap is achieved, thereby stabilizing the likelihood anchor of the preferred output—be it visual tokens in VLMs or evidence-trace branches in agent trees. This targeted asymmetry addresses core weaknesses of symmetric DPO objectives, yielding measurable increases in factual accuracy and grounding in both single- and multi-model settings. Empirical demonstrations on InternVL3 and across multi-agent math reasoning frameworks substantiate the practical utility of ACPO, which remains theoretically generalizable and computationally accessible for a range of future alignment and consensus-adjudication problems (Huang et al., 23 Mar 2026, Yang et al., 10 Feb 2026).