---
title: Bias-Subtracted Contrastive Decoding
url: https://www.emergentmind.com/topics/bias-subtracted-contrastive-decoding
type: topic
---

# Bias-Subtracted Contrastive Decoding

Bias-Subtracted Contrastive Decoding denotes a family of inference-time decoding methods that alter token- or candidate-level scores by subtracting a distribution associated with an undesired prior from the task-conditioned distribution. The bias term may be induced by a perturbed textual input, a distorted visual input, a source-only continuation prior, an irrelevant retrieved passage, a smaller assistant model from the same family, or a text-only answer-choice prior. Across LLMs and LVLMs, the common objective is to prefer outputs that are likely under the intended conditioning and unlikely under a bias-inducing condition, without modifying model parameters. This family includes Contrastive Input Decoding (CID), Visual Contrastive Decoding (VCD), Anti-LM decoding, tri-distribution context-grounded decoding for open-domain QA, SCICON for scientific figure MCQA, SDCD for LVLM hallucination mitigation, Speculative Contrastive Decoding, and contrastive decoding for score-range bias in LLM-as-a-judge [2305.07378] [2311.16922] [2311.08324].

## 1. Conceptual core and mathematical forms

The central mechanism is score subtraction at decoding time. In its most explicit generic form, Anti-LM writes the adjusted next-token score as
$$
s(y \mid x,u,y_{<t}) = \log p_M(y \mid C_{\mathrm{full}}(x,u,y_{<t})) - \lambda_t \log p_A(y \mid C_{\mathrm{bias}}(x)),
$$
where the task context and the bias context differ by construction. In Anti-LM, the bias model is the same network run on source-only context, so the subtracted term is the model’s tendency to continue the source sentence rather than translate it [2311.08324].

CID introduces a probability-difference formulation for two inputs, an original input $x$ and a contrastive input $x'$. At each decoding step it defines
$$
A(w; x, x') = P_M(w \mid x_{\mathrm{pre}}) - P_M(w \mid x'_{\mathrm{pre}})
$$
and reweights the original distribution by $\exp(\lambda A(w; x, x'))$, yielding the greedy score
$$
s(w \mid x, x') = \log P_M(w \mid x_{\mathrm{pre}}) + \lambda \cdot A(w; x, x').
$$
This formulation favors tokens that are more likely under the original input than under the perturbed input and suppresses tokens that the perturbation makes more likely [2305.07378].

A second major branch operates directly in logit space. VCD defines
$$
p_{\mathrm{vcd}}(y \mid v,v',x)=\operatorname{softmax}\Big((1+\alpha)\,\operatorname{logit}_\theta(y \mid v,x)-\alpha\,\operatorname{logit}_\theta(y \mid v',x)\Big),
$$
where $v$ is the original image and $v'$ is a distorted image. SDCD uses the same structural form with a structure-disrupted image $I^{sd}$:
$$
s_t=(1+\alpha)\,z_t(I,x_{<t})-\alpha\,z_t(I^{sd},x_{<t}).
$$
Both methods can be read as bias subtraction in logit space: tokens that remain probable under the negative view are penalized, whereas tokens supported by intact visual evidence are preserved [2311.16922] [2601.03500].

A further extension is the tri-distribution form for context-grounded QA:
$$
s_t(y)=z_t(y)+\alpha_t\big(z_t^{+}(y)-z_t^{-}(y)\big),
$$
where $z_t$ is the prior or query-only logit, $z_t^{+}$ is the relevant-context logit, and $z_t^{-}$ is the irrelevant-context logit. Here the prior is not discarded; it is retained as a stabilizing base while the relevant context is amplified and the irrelevant context is subtracted [2405.02750].

Taken together, these formulations define a common family: decoding is steered by a ratio, difference, or affine contrast between a task-conditioned signal and a bias-conditioned signal. This suggests that “bias subtraction” is not tied to a single modality or objective; it is a decoding template whose semantics depend on how the negative view is constructed.

## 2. Decoding procedure, plausibility constraints, and computational profile

Most instantiations apply contrast at every decoding step. VCD computes visual embeddings for $v$ and $v'$, maintains two KV-caches, forms contrastive logits at each step, applies an adaptive plausibility constraint with threshold $\beta$, and then samples or decodes greedily from the adjusted distribution. The candidate set is
$$
\mathcal{V}_{\mathrm{head}(y_{<t})}=\{y_t \in \mathcal{V}: p_\theta(y_t \mid v,x,y_{<t}) \ge \beta \cdot \max_w p_\theta(w \mid v,x,y_{<t})\},
$$
with $p_{\mathrm{vcd}}(y_t)=0$ outside this set. The method requires approximately $2\times$ language forward passes per decoding step if computed separately, though batching the two contexts reduces wall-clock penalty [2311.16922].

CID uses a related but not identical restriction mechanism. In all reported experiments it truncates the original distribution to top-$K$ before applying the multiplicative contrastive reweighting, with $K=50$ throughout. The shared generated prefix is appended to both contexts at each step so that the two contexts “do not continuously diverge, but always differ only in the ways the original and contrastive inputs differ.” Its default decoding mode is greedy, and its complexity is roughly $2\times$ that of standard decoding, with batched inference and dual caches as the main optimization route [2305.07378].

Anti-LM is computationally distinct because the bias vector is precomputed once per source sentence:
$$
b=\log p_M(\cdot \mid X).
$$
At time step $t$, the decoder uses
$$
s_t = m_t - \lambda_t \cdot b,
$$
with $\lambda_t=\lambda_0^t$ and $\lambda_0=0.3$ in the reported experiments. This yields one extra forward pass per source sentence rather than one extra pass per token, so its overhead is substantially smaller than per-step dual-view methods [2311.08324].

The open-domain QA method with relevant and irrelevant passages requires three forward passes per step, corresponding to the prior, positive-context, and negative-context branches, and therefore incurs roughly $3\times$ decoding cost. It introduces a dynamic token-level $\alpha_t$ based on confidence comparison between prior and relevant-context distributions, rather than fixing a global contrast weight [2405.02750].

Speculative Contrastive Decoding modifies the computational trade-off. It uses a smaller amateur LM to draft $\gamma$ tokens, then verifies them against a contrastive target distribution defined by the expert and amateur models. The acceptance rule is
$$
r_i \le \frac{P_n^\tau(x_i \mid x_{<i})}{P_{M_a}(x_i \mid x_{<i})},
$$
and rejected tokens are resampled from a residual contrastive distribution. The paper reports expected acceleration factors, using $c=0.05$, of $\times 1.78$ to $\times 3.32$ for original SCD and $\times 2.10$ to $\times 3.32$ for improved SCD, while matching the quality improvements of standard contrastive decoding [2311.08981].

## 3. Negative-view design and the varieties of bias being subtracted

The defining design choice in Bias-Subtracted Contrastive Decoding is the construction of the negative view. Different papers operationalize different notions of “bias,” and the effectiveness of the method depends on whether the negative view isolates the relevant failure mode.

| Method | Negative view or bias term | Target failure mode |
|---|---|---|
| CID [2305.07378] | Perturbed textual input $x'$ | Context-specific bias surfacing |
| VCD [2311.16922] | Distorted image $v'$ via Gaussian noise | Object hallucination from statistical bias and unimodal priors |
| Anti-LM [2311.08324] | Source-only prior $p_M(\cdot \mid X)$ | Failure to translate in zero-shot MT |
| Context-grounded QA [2405.02750] | Irrelevant or adversarial passage $c^{-}$ | Over-reliance on irrelevant context or parametric prior |
| SCICON [2603.28026] | Text-only candidate logits | Choice-induced prior bias in scientific MCQA |
| SDCD [2601.03500] | Structure-disrupted image $I^{sd}$ from patch shuffling | Texture-driven visual statistical bias |
| Judge CD [2510.18196] | Same-family assistant model probabilities | Score range bias in direct assessment |

In CID, the perturbation is intentionally minimal and chosen to isolate a suspected bias axis, such as demographic swaps or framing changes. The authors emphasize that the method is designed to surface biases that standard decoding can hide, not primarily to alter the base model’s training-time behavior [2305.07378].

VCD uses Gaussian-noised versions of an image as the bias view. Its rationale is that increasing visual uncertainty amplifies language priors and corpus biases; contrasting the original and distorted inputs then emphasizes visual evidence that survives only under the intact image. The paper notes that stronger distortions degrade POPE performance under regular decoding, which it interprets as evidence that visual uncertainty amplifies unimodal priors [2311.16922].

SDCD targets a different source of error. Instead of adding pixel noise, it shuffles image patches to preserve local texture statistics while destroying global geometry. This negative view is tailored to the “Bag-of-Patches” tendency of transformer vision encoders under weak structural supervision. The paper argues that hallucinated tokens often remain confident under the shuffled view, whereas real object tokens lose confidence, a pattern it terms Structure Sensitivity Divergence [2601.03500].

SCICON removes the image entirely and subtracts text-only candidate logits from image-conditioned candidate logits:
$$
l_{\mathrm{sc}}(c)=l_{\mathrm{mm}}(c)-\alpha\,l_{\mathrm{txt}}(c).
$$
Here the bias is not a hallucinated visual object or a source-language prior, but the scientifically plausible phrasing of answer choices themselves [2603.28026].

The score-range-bias method for LLM-as-a-judge uses a smaller assistant from the same model family, run on the same prompt and same allowed label set. The assumption is that family-shared priors over score ranges are present in both models, often more strongly in the smaller one, so subtracting assistant probabilities cancels the shared range-induced skew [2510.18196].

## 4. Representative instantiations and empirical evidence

The best-developed multimodal instance is VCD. On POPE, VCD reports consistent gains across LLaVA-1.5, Qwen-VL, and InstructBLIP. Representative improvements on MSCOCO Random include LLaVA-1.5 F1 $81.33 \rightarrow 87.16$ and Accuracy $83.29 \rightarrow 87.73$, Qwen-VL F1 $82.67 \rightarrow 87.81$ and Accuracy $84.73 \rightarrow 88.63$, and InstructBLIP F1 $80.41 \rightarrow 83.68$ and Accuracy $80.71 \rightarrow 84.53$. On the MME hallucination subset, LLaVA-1.5 total score increases from $565.33$ to $604.66$, Qwen-VL from $587.33$ to $596.67$, and InstructBLIP from $380.33$ to $447.67$. On LLaVA-Bench, both Accuracy and Detailedness improve, including Qwen-VL Accuracy $4.76 \rightarrow 6.69$ and Detailedness $3.46 \rightarrow 4.46$ [2311.16922].

SDCD reports further improvements by targeting structure-less texture bias. On POPE with LLaVA-1.5, MSCOCO Random, Regular decoding yields Acc $82.93$ and F1 $80.87$, VCD yields Acc $84.87$ and F1 $83.37$, and SDCD yields Acc $85.90$ and F1 $84.56$. For Qwen2.5-VL on the same split, SDCD raises recall from $64.53$ under Regular and $66.13$ under VCD to $72.07$, with F1 reaching $83.38$. On MME with the LLaVA-1.5 backbone, SDCD raises Perception total to $1348.35$ and Cognition total to $338.93$, compared with $1292.01$ and $286.43$ for VCD. On CHAIR, it reports CHAIR\_S $18.6$ and CHAIR\_I $6.4$, markedly lower than Regular and VCD [2601.03500].

In zero-shot in-context machine translation, Anti-LM directly subtracts the source-continuation prior. Under beam search with $B=5$, selected SacreBLEU improvements include XGLM-2.9B on en$\rightarrow$fr $17.9 \rightarrow 26.6$, XGLM-7.5B on en$\rightarrow$pt $3.7 \rightarrow 27.0$, and XGLM-7.5B on en$\rightarrow$de $4.1 \rightarrow 18.5$. The paper summarizes the effect as “up to $20$ BLEU point improvement” in some settings, with the largest gains occurring where default decoding strongly fails to translate [2311.08324].

For scientific figure multiple-choice QA, SCICON consistently improves accuracy over greedy decoding and typically surpasses VCD and ICD. With Qwen 3.5 4B, MAC accuracy rises from $69.72$ to $74.01$, SciFIBench from $46.20$ to $48.70$, and MMSci from $38.83$ to $43.44$. With Qwen 3.5 9B, MMSci improves from $46.54$ to $52.14$. With Phi-3.5-vision-instruct, MAC increases from $42.81$ to $49.54$ and SciFIBench from $48.60$ to $54.90$ [2603.28026].

The open-domain QA tri-distribution method reports Exact Match gains over regular open-book decoding and CAD. For Llama 2 70B, Regular yields NQ $56.07$, TriviaQA $76.07$, and PopQA $42.7$; the fixed-$\alpha$ variant yields $58.86$, $78.38$, and $42.59$, while the dynamic-$\alpha$ variant yields $55.24$, $81.7$, and $44.3$. On Flan-T5 11B, the dynamic variant raises NQ from $57.84$ to $63.16$ and PopQA from $31.16$ to $34.64$ [2405.02750].

Speculative Contrastive Decoding demonstrates that bias subtraction can coexist with acceleration. On AlpacaEval, SCD\_imp reports a GPT-4 judged win-rate of $95.03$, compared with $94.78$ for CD\_imp and $94.66$ for the expert model alone. On GSM8k, SCD\_imp attains Accuracy $64.90$, matching CD\_imp’s $64.91$ while retaining expected acceleration. On HumanEval, SCD\_ori reaches Pass@1 $37.20$, equal to CD\_ori and above the expert model’s $28.66$ [2311.08981].

In LLM-as-a-judge, same-family contrastive subtraction mitigates score range bias. For coherence, Llama-3.1-8B-Instruct improves average Spearman from $0.334$ to $0.352$ with a 1B assistant, while Qwen-2.5-14B-Instruct improves from $0.384$ to $0.433$ with a 3B assistant. The paper reports up to $11.3\%$ relative improvement on average in Spearman correlation with human judgments across score ranges [2510.18196].

## 5. Relation to adjacent methods and recurring misconceptions

Bias-Subtracted Contrastive Decoding is often grouped with “contrastive decoding,” but the family is heterogeneous. Classical expert-versus-amateur contrastive decoding contrasts two models on the same input, whereas CID contrasts two inputs with the same model and is explicitly framed as a way to surface differences that standard decoding misses [2305.07378]. Anti-LM, by contrast, does not contrast two models or two prompts in the usual sense; it contrasts full translation context against a source-only continuation prior and is positioned as a calibration method for zero-shot MT [2311.08324].

In LVLMs, VCD and SDCD differ in what they assume the dominant bias to be. VCD targets over-reliance on statistical bias and unimodal priors using Gaussian-noised images, whereas SDCD targets visual statistical bias arising from the encoder’s Bag-of-Patches behavior using shuffled patch structure. The papers present these methods as orthogonal in emphasis: one subtracts a visually uncertain prior, the other subtracts a structure-less texture prior [2311.16922] [2601.03500].

The open-domain QA method is also distinct from CAD-style formulations that subtract a no-context distribution. It retains the prior as an additive base and subtracts only the irrelevant-context branch. This suggests a different interpretation of bias subtraction: not all unwanted signal comes from parametric memory; some comes from misleading non-parametric context [2405.02750].

A recurring misconception is that contrastive subtraction automatically “debiases” the model. CID explicitly warns against this reading: it is designed primarily as an auditing and diagnostic tool, and if used for mitigation it may create superficial changes that mask deeper issues [2305.07378]. Similarly, score-range mitigation in LLM-as-a-judge changes the decision boundary over allowed labels before generation; it does not retrain the judge or remove family-level priors from the underlying model weights [2510.18196].

Another misconception is that the negative branch must always be unconditional or text-only. The surveyed literature shows otherwise: the negative branch may be a distorted image, a shuffled image, a source-only prior, a contrastive prompt, an irrelevant passage, or an assistant model. What unifies the family is not the modality of the negative branch, but the use of subtractive decoding to suppress outputs that remain likely under an undesirable condition.

## 6. Limitations, failure modes, and open directions

Across papers, the dominant failure mode is over-correction. VCD notes that large $\alpha$ or heavy distortion can prune too aggressively, reducing descriptive richness or causing brittle outputs; its plausibility constraint with $\beta$ is intended to mitigate this [2311.16922]. SDCD reports a similar precision–recall trade-off under excessive $\alpha$, and also finds that overly coarse shuffling weakens the structural contrast signal [2601.03500]. CID warns that overly large $\lambda$ can produce degenerate or incoherent outputs and that the quality of insights depends on the quality of the perturbation $x_c$ [2305.07378].

The choice of negative view is itself a source of brittleness. In the contextual QA method, if the irrelevant passage $c^{-}$ accidentally contains relevant information, subtraction can harm performance; the paper therefore finds “most distant” negatives preferable to random ones [2405.02750]. SCICON reports that when the figure adds little beyond the text prior, subtracting the text-only branch can remove useful information and flip correct predictions to errors [2603.28026]. The LLM-as-a-judge method likewise depends on selecting a same-family assistant and tuning $\lambda$ and assistant temperature $t$; poorly tuned assistants can over-impose family bias rather than cancel it [2510.18196].

Some limitations are architecture- or domain-specific. SDCD works best when the visual projector preserves patch-level independence, with gains smaller or mixed for Resampler-based projectors [2601.03500]. SCICON is evaluated only in MCQA over scientific figures, and its extension to open-ended multimodal reasoning is described as unverified [2603.28026]. The score-range-bias study is limited to English summarization tasks and models up to $14$B parameters [2510.18196].

Compute remains a recurring cost. Dual-view methods typically require two forward passes per step, and the tri-distribution QA method requires three. Anti-LM is an exception because it computes its bias term once per source sentence [2311.08324]. SCD shows that contrastive subtraction can be made compatible with acceleration, but its benefits depend on acceptance rates, model cost ratio $c$, and serving infrastructure [2311.08981].

The open directions are correspondingly diverse. VCD proposes richer distortions such as object-level masking, region blurring, color jitter, or occlusions, and extension to multi-image, multi-turn dialogue, and spatial-reasoning tasks [2311.16922]. CID suggests multi-contrast inputs, adaptive $\lambda$, and span-level contrast [2305.07378]. The contextual QA method suggests multiple negatives and alternative aggregation rules [2405.02750]. SCICON points toward adaptive weighting based on prior alignment or confidence [2603.28026]. Taken together, these proposals indicate that the future of Bias-Subtracted Contrastive Decoding lies less in a single canonical formula than in better design of bias proxies, adaptive weighting schemes, and task-specific negative views.

Source: https://www.emergentmind.com/topics/bias-subtracted-contrastive-decoding