Bias-Subtracted Contrastive Decoding

• Bias-Subtracted Contrastive Decoding is a family of inference methods that subtracts an undesired bias distribution from the task-conditioned score to improve output relevance.
• It leverages contrastive inputs—such as perturbed text, distorted images, or assistant model outputs—to suppress statistical biases without modifying model parameters.
• Empirical evaluations show significant gains in metrics across tasks like translation, QA, and hallucination mitigation, despite increased per-step computational cost.

Bias-Subtracted Contrastive Decoding denotes a family of inference-time decoding methods that alter token- or candidate-level scores by subtracting a distribution associated with an undesired prior from the task-conditioned distribution. The bias term may be induced by a perturbed textual input, a distorted visual input, a source-only continuation prior, an irrelevant retrieved passage, a smaller assistant model from the same family, or a text-only answer-choice prior. Across LLMs and LVLMs, the common objective is to prefer outputs that are likely under the intended conditioning and unlikely under a bias-inducing condition, without modifying model parameters. This family includes Contrastive Input Decoding (CID), Visual Contrastive Decoding (VCD), Anti-LM decoding, tri-distribution context-grounded decoding for open-domain QA, SCICON for scientific figure MCQA, SDCD for LVLM hallucination mitigation, Speculative Contrastive Decoding, and contrastive decoding for score-range bias in LLM-as-a-judge (Yona et al., 2023, Leng et al., 2023, Sia et al., 2023).

1. Conceptual core and mathematical forms

The central mechanism is score subtraction at decoding time. In its most explicit generic form, Anti-LM writes the adjusted next-token score as

$$s(y \mid x,u,y_{<t}) = \log p_M(y \mid C_{\mathrm{full}}(x,u,y_{<t})) - \lambda_t \log p_A(y \mid C_{\mathrm{bias}}(x)),$$

where the task context and the bias context differ by construction. In Anti-LM, the bias model is the same network run on source-only context, so the subtracted term is the model’s tendency to continue the source sentence rather than translate it (Sia et al., 2023).

CID introduces a probability-difference formulation for two inputs, an original input $x$ and a contrastive input $x'$. At each decoding step it defines

$$A(w; x, x') = P_M(w \mid x_{\mathrm{pre}}) - P_M(w \mid x'_{\mathrm{pre}})$$

and reweights the original distribution by $\exp(\lambda A(w; x, x'))$, yielding the greedy score

$$s(w \mid x, x') = \log P_M(w \mid x_{\mathrm{pre}}) + \lambda \cdot A(w; x, x').$$

This formulation favors tokens that are more likely under the original input than under the perturbed input and suppresses tokens that the perturbation makes more likely (Yona et al., 2023).

A second major branch operates directly in logit space. VCD defines

$$p_{\mathrm{vcd}}(y \mid v,v',x)=\operatorname{softmax}\Big((1+\alpha)\,\operatorname{logit}_\theta(y \mid v,x)-\alpha\,\operatorname{logit}_\theta(y \mid v',x)\Big),$$

where $v$ is the original image and $v'$ is a distorted image. SDCD uses the same structural form with a structure-disrupted image $I^{sd}$:

$x$0

Both methods can be read as bias subtraction in logit space: tokens that remain probable under the negative view are penalized, whereas tokens supported by intact visual evidence are preserved (Leng et al., 2023, Xia et al., 7 Jan 2026).

A further extension is the tri-distribution form for context-grounded QA:

$x$1

where $x$2 is the prior or query-only logit, $x$3 is the relevant-context logit, and $x$4 is the irrelevant-context logit. Here the prior is not discarded; it is retained as a stabilizing base while the relevant context is amplified and the irrelevant context is subtracted (Zhao et al., 2024).

Taken together, these formulations define a common family: decoding is steered by a ratio, difference, or affine contrast between a task-conditioned signal and a bias-conditioned signal. This suggests that “bias subtraction” is not tied to a single modality or objective; it is a decoding template whose semantics depend on how the negative view is constructed.

2. Decoding procedure, plausibility constraints, and computational profile

Most instantiations apply contrast at every decoding step. VCD computes visual embeddings for $x$5 and $x$6, maintains two KV-caches, forms contrastive logits at each step, applies an adaptive plausibility constraint with threshold $x$7, and then samples or decodes greedily from the adjusted distribution. The candidate set is

$x$8

with $x$9 outside this set. The method requires approximately $x'$0 language forward passes per decoding step if computed separately, though batching the two contexts reduces wall-clock penalty (Leng et al., 2023).

CID uses a related but not identical restriction mechanism. In all reported experiments it truncates the original distribution to top-$x'$1 before applying the multiplicative contrastive reweighting, with $x'$2 throughout. The shared generated prefix is appended to both contexts at each step so that the two contexts “do not continuously diverge, but always differ only in the ways the original and contrastive inputs differ.” Its default decoding mode is greedy, and its complexity is roughly $x'$3 that of standard decoding, with batched inference and dual caches as the main optimization route (Yona et al., 2023).

Anti-LM is computationally distinct because the bias vector is precomputed once per source sentence:

$x'$4

At time step $x'$5, the decoder uses

$x'$6

with $x'$7 and $x'$8 in the reported experiments. This yields one extra forward pass per source sentence rather than one extra pass per token, so its overhead is substantially smaller than per-step dual-view methods (Sia et al., 2023).

The open-domain QA method with relevant and irrelevant passages requires three forward passes per step, corresponding to the prior, positive-context, and negative-context branches, and therefore incurs roughly $x'$9 decoding cost. It introduces a dynamic token-level $$A(w; x, x') = P_M(w \mid x_{\mathrm{pre}}) - P_M(w \mid x'_{\mathrm{pre}})$$0 based on confidence comparison between prior and relevant-context distributions, rather than fixing a global contrast weight (Zhao et al., 2024).

Speculative Contrastive Decoding modifies the computational trade-off. It uses a smaller amateur LM to draft $$A(w; x, x') = P_M(w \mid x_{\mathrm{pre}}) - P_M(w \mid x'_{\mathrm{pre}})$$1 tokens, then verifies them against a contrastive target distribution defined by the expert and amateur models. The acceptance rule is

$$A(w; x, x') = P_M(w \mid x_{\mathrm{pre}}) - P_M(w \mid x'_{\mathrm{pre}})$$2

and rejected tokens are resampled from a residual contrastive distribution. The paper reports expected acceleration factors, using $$A(w; x, x') = P_M(w \mid x_{\mathrm{pre}}) - P_M(w \mid x'_{\mathrm{pre}})$$3, of $$A(w; x, x') = P_M(w \mid x_{\mathrm{pre}}) - P_M(w \mid x'_{\mathrm{pre}})$$4 to $$A(w; x, x') = P_M(w \mid x_{\mathrm{pre}}) - P_M(w \mid x'_{\mathrm{pre}})$$5 for original SCD and $$A(w; x, x') = P_M(w \mid x_{\mathrm{pre}}) - P_M(w \mid x'_{\mathrm{pre}})$$6 to $$A(w; x, x') = P_M(w \mid x_{\mathrm{pre}}) - P_M(w \mid x'_{\mathrm{pre}})$$7 for improved SCD, while matching the quality improvements of standard contrastive decoding (Yuan et al., 2023).

3. Negative-view design and the varieties of bias being subtracted

The defining design choice in Bias-Subtracted Contrastive Decoding is the construction of the negative view. Different papers operationalize different notions of “bias,” and the effectiveness of the method depends on whether the negative view isolates the relevant failure mode.

Method	Negative view or bias term	Target failure mode
CID (Yona et al., 2023)	Perturbed textual input $$A(w; x, x') = P_M(w \mid x_{\mathrm{pre}}) - P_M(w \mid x'_{\mathrm{pre}})$$8	Context-specific bias surfacing
VCD (Leng et al., 2023)	Distorted image $$A(w; x, x') = P_M(w \mid x_{\mathrm{pre}}) - P_M(w \mid x'_{\mathrm{pre}})$$9 via Gaussian noise	Object hallucination from statistical bias and unimodal priors
Anti-LM (Sia et al., 2023)	Source-only prior $\exp(\lambda A(w; x, x'))$0	Failure to translate in zero-shot MT
Context-grounded QA (Zhao et al., 2024)	Irrelevant or adversarial passage $\exp(\lambda A(w; x, x'))$1	Over-reliance on irrelevant context or parametric prior
SCICON (Roh et al., 30 Mar 2026)	Text-only candidate logits	Choice-induced prior bias in scientific MCQA
SDCD (Xia et al., 7 Jan 2026)	Structure-disrupted image $\exp(\lambda A(w; x, x'))$2 from patch shuffling	Texture-driven visual statistical bias
Judge CD (Fujinuma, 21 Oct 2025)	Same-family assistant model probabilities	Score range bias in direct assessment

In CID, the perturbation is intentionally minimal and chosen to isolate a suspected bias axis, such as demographic swaps or framing changes. The authors emphasize that the method is designed to surface biases that standard decoding can hide, not primarily to alter the base model’s training-time behavior (Yona et al., 2023).

VCD uses Gaussian-noised versions of an image as the bias view. Its rationale is that increasing visual uncertainty amplifies language priors and corpus biases; contrasting the original and distorted inputs then emphasizes visual evidence that survives only under the intact image. The paper notes that stronger distortions degrade POPE performance under regular decoding, which it interprets as evidence that visual uncertainty amplifies unimodal priors (Leng et al., 2023).

SDCD targets a different source of error. Instead of adding pixel noise, it shuffles image patches to preserve local texture statistics while destroying global geometry. This negative view is tailored to the “Bag-of-Patches” tendency of transformer vision encoders under weak structural supervision. The paper argues that hallucinated tokens often remain confident under the shuffled view, whereas real object tokens lose confidence, a pattern it terms Structure Sensitivity Divergence (Xia et al., 7 Jan 2026).

SCICON removes the image entirely and subtracts text-only candidate logits from image-conditioned candidate logits:

$\exp(\lambda A(w; x, x'))$3

Here the bias is not a hallucinated visual object or a source-language prior, but the scientifically plausible phrasing of answer choices themselves (Roh et al., 30 Mar 2026).

The score-range-bias method for LLM-as-a-judge uses a smaller assistant from the same model family, run on the same prompt and same allowed label set. The assumption is that family-shared priors over score ranges are present in both models, often more strongly in the smaller one, so subtracting assistant probabilities cancels the shared range-induced skew (Fujinuma, 21 Oct 2025).

4. Representative instantiations and empirical evidence

The best-developed multimodal instance is VCD. On POPE, VCD reports consistent gains across LLaVA-1.5, Qwen-VL, and InstructBLIP. Representative improvements on MSCOCO Random include LLaVA-1.5 F1 $\exp(\lambda A(w; x, x'))$4 and Accuracy $\exp(\lambda A(w; x, x'))$5, Qwen-VL F1 $\exp(\lambda A(w; x, x'))$6 and Accuracy $\exp(\lambda A(w; x, x'))$7, and InstructBLIP F1 $\exp(\lambda A(w; x, x'))$8 and Accuracy $\exp(\lambda A(w; x, x'))$9. On the MME hallucination subset, LLaVA-1.5 total score increases from $$s(w \mid x, x') = \log P_M(w \mid x_{\mathrm{pre}}) + \lambda \cdot A(w; x, x').$$0 to $$s(w \mid x, x') = \log P_M(w \mid x_{\mathrm{pre}}) + \lambda \cdot A(w; x, x').$$1, Qwen-VL from $$s(w \mid x, x') = \log P_M(w \mid x_{\mathrm{pre}}) + \lambda \cdot A(w; x, x').$$2 to $$s(w \mid x, x') = \log P_M(w \mid x_{\mathrm{pre}}) + \lambda \cdot A(w; x, x').$$3, and InstructBLIP from $$s(w \mid x, x') = \log P_M(w \mid x_{\mathrm{pre}}) + \lambda \cdot A(w; x, x').$$4 to $$s(w \mid x, x') = \log P_M(w \mid x_{\mathrm{pre}}) + \lambda \cdot A(w; x, x').$$5. On LLaVA-Bench, both Accuracy and Detailedness improve, including Qwen-VL Accuracy $$s(w \mid x, x') = \log P_M(w \mid x_{\mathrm{pre}}) + \lambda \cdot A(w; x, x').$$6 and Detailedness $$s(w \mid x, x') = \log P_M(w \mid x_{\mathrm{pre}}) + \lambda \cdot A(w; x, x').$$7 (Leng et al., 2023).

SDCD reports further improvements by targeting structure-less texture bias. On POPE with LLaVA-1.5, MSCOCO Random, Regular decoding yields Acc $$s(w \mid x, x') = \log P_M(w \mid x_{\mathrm{pre}}) + \lambda \cdot A(w; x, x').$$8 and F1 $$s(w \mid x, x') = \log P_M(w \mid x_{\mathrm{pre}}) + \lambda \cdot A(w; x, x').$$9, VCD yields Acc $$p_{\mathrm{vcd}}(y \mid v,v',x)=\operatorname{softmax}\Big((1+\alpha)\,\operatorname{logit}_\theta(y \mid v,x)-\alpha\,\operatorname{logit}_\theta(y \mid v',x)\Big),$$0 and F1 $$p_{\mathrm{vcd}}(y \mid v,v',x)=\operatorname{softmax}\Big((1+\alpha)\,\operatorname{logit}_\theta(y \mid v,x)-\alpha\,\operatorname{logit}_\theta(y \mid v',x)\Big),$$1, and SDCD yields Acc $$p_{\mathrm{vcd}}(y \mid v,v',x)=\operatorname{softmax}\Big((1+\alpha)\,\operatorname{logit}_\theta(y \mid v,x)-\alpha\,\operatorname{logit}_\theta(y \mid v',x)\Big),$$2 and F1 $$p_{\mathrm{vcd}}(y \mid v,v',x)=\operatorname{softmax}\Big((1+\alpha)\,\operatorname{logit}_\theta(y \mid v,x)-\alpha\,\operatorname{logit}_\theta(y \mid v',x)\Big),$$3. For Qwen2.5-VL on the same split, SDCD raises recall from $$p_{\mathrm{vcd}}(y \mid v,v',x)=\operatorname{softmax}\Big((1+\alpha)\,\operatorname{logit}_\theta(y \mid v,x)-\alpha\,\operatorname{logit}_\theta(y \mid v',x)\Big),$$4 under Regular and $$p_{\mathrm{vcd}}(y \mid v,v',x)=\operatorname{softmax}\Big((1+\alpha)\,\operatorname{logit}_\theta(y \mid v,x)-\alpha\,\operatorname{logit}_\theta(y \mid v',x)\Big),$$5 under VCD to $$p_{\mathrm{vcd}}(y \mid v,v',x)=\operatorname{softmax}\Big((1+\alpha)\,\operatorname{logit}_\theta(y \mid v,x)-\alpha\,\operatorname{logit}_\theta(y \mid v',x)\Big),$$6, with F1 reaching $$p_{\mathrm{vcd}}(y \mid v,v',x)=\operatorname{softmax}\Big((1+\alpha)\,\operatorname{logit}_\theta(y \mid v,x)-\alpha\,\operatorname{logit}_\theta(y \mid v',x)\Big),$$7. On MME with the LLaVA-1.5 backbone, SDCD raises Perception total to $$p_{\mathrm{vcd}}(y \mid v,v',x)=\operatorname{softmax}\Big((1+\alpha)\,\operatorname{logit}_\theta(y \mid v,x)-\alpha\,\operatorname{logit}_\theta(y \mid v',x)\Big),$$8 and Cognition total to $$p_{\mathrm{vcd}}(y \mid v,v',x)=\operatorname{softmax}\Big((1+\alpha)\,\operatorname{logit}_\theta(y \mid v,x)-\alpha\,\operatorname{logit}_\theta(y \mid v',x)\Big),$$9, compared with $v$0 and $v$1 for VCD. On CHAIR, it reports CHAIR_S $v$2 and CHAIR_I $v$3, markedly lower than Regular and VCD (Xia et al., 7 Jan 2026).

In zero-shot in-context machine translation, Anti-LM directly subtracts the source-continuation prior. Under beam search with $v$4, selected SacreBLEU improvements include XGLM-2.9B on en$v$5fr $v$6, XGLM-7.5B on en$v$7pt $v$8, and XGLM-7.5B on en$v$9de $v'$0. The paper summarizes the effect as “up to $v'$1 BLEU point improvement” in some settings, with the largest gains occurring where default decoding strongly fails to translate (Sia et al., 2023).

For scientific figure multiple-choice QA, SCICON consistently improves accuracy over greedy decoding and typically surpasses VCD and ICD. With Qwen 3.5 4B, MAC accuracy rises from $v'$2 to $v'$3, SciFIBench from $v'$4 to $v'$5, and MMSci from $v'$6 to $v'$7. With Qwen 3.5 9B, MMSci improves from $v'$8 to $v'$9. With Phi-3.5-vision-instruct, MAC increases from $I^{sd}$0 to $I^{sd}$1 and SciFIBench from $I^{sd}$2 to $I^{sd}$3 (Roh et al., 30 Mar 2026).

The open-domain QA tri-distribution method reports Exact Match gains over regular open-book decoding and CAD. For Llama 2 70B, Regular yields NQ $I^{sd}$4, TriviaQA $I^{sd}$5, and PopQA $I^{sd}$6; the fixed-$I^{sd}$7 variant yields $I^{sd}$8, $I^{sd}$9, and $x$00, while the dynamic-$x$01 variant yields $x$02, $x$03, and $x$04. On Flan-T5 11B, the dynamic variant raises NQ from $x$05 to $x$06 and PopQA from $x$07 to $x$08 (Zhao et al., 2024).

Speculative Contrastive Decoding demonstrates that bias subtraction can coexist with acceleration. On AlpacaEval, SCD_imp reports a GPT-4 judged win-rate of $x$09, compared with $x$10 for CD_imp and $x$11 for the expert model alone. On GSM8k, SCD_imp attains Accuracy $x$12, matching CD_imp’s $x$13 while retaining expected acceleration. On HumanEval, SCD_ori reaches Pass@1 $x$14, equal to CD_ori and above the expert model’s $x$15 (Yuan et al., 2023).

In LLM-as-a-judge, same-family contrastive subtraction mitigates score range bias. For coherence, Llama-3.1-8B-Instruct improves average Spearman from $x$16 to $x$17 with a 1B assistant, while Qwen-2.5-14B-Instruct improves from $x$18 to $x$19 with a 3B assistant. The paper reports up to $x$20 relative improvement on average in Spearman correlation with human judgments across score ranges (Fujinuma, 21 Oct 2025).

5. Relation to adjacent methods and recurring misconceptions

Bias-Subtracted Contrastive Decoding is often grouped with “contrastive decoding,” but the family is heterogeneous. Classical expert-versus-amateur contrastive decoding contrasts two models on the same input, whereas CID contrasts two inputs with the same model and is explicitly framed as a way to surface differences that standard decoding misses (Yona et al., 2023). Anti-LM, by contrast, does not contrast two models or two prompts in the usual sense; it contrasts full translation context against a source-only continuation prior and is positioned as a calibration method for zero-shot MT (Sia et al., 2023).

In LVLMs, VCD and SDCD differ in what they assume the dominant bias to be. VCD targets over-reliance on statistical bias and unimodal priors using Gaussian-noised images, whereas SDCD targets visual statistical bias arising from the encoder’s Bag-of-Patches behavior using shuffled patch structure. The papers present these methods as orthogonal in emphasis: one subtracts a visually uncertain prior, the other subtracts a structure-less texture prior (Leng et al., 2023, Xia et al., 7 Jan 2026).

The open-domain QA method is also distinct from CAD-style formulations that subtract a no-context distribution. It retains the prior as an additive base and subtracts only the irrelevant-context branch. This suggests a different interpretation of bias subtraction: not all unwanted signal comes from parametric memory; some comes from misleading non-parametric context (Zhao et al., 2024).

A recurring misconception is that contrastive subtraction automatically “debiases” the model. CID explicitly warns against this reading: it is designed primarily as an auditing and diagnostic tool, and if used for mitigation it may create superficial changes that mask deeper issues (Yona et al., 2023). Similarly, score-range mitigation in LLM-as-a-judge changes the decision boundary over allowed labels before generation; it does not retrain the judge or remove family-level priors from the underlying model weights (Fujinuma, 21 Oct 2025).

Another misconception is that the negative branch must always be unconditional or text-only. The surveyed literature shows otherwise: the negative branch may be a distorted image, a shuffled image, a source-only prior, a contrastive prompt, an irrelevant passage, or an assistant model. What unifies the family is not the modality of the negative branch, but the use of subtractive decoding to suppress outputs that remain likely under an undesirable condition.

6. Limitations, failure modes, and open directions

Across papers, the dominant failure mode is over-correction. VCD notes that large $x$21 or heavy distortion can prune too aggressively, reducing descriptive richness or causing brittle outputs; its plausibility constraint with $x$22 is intended to mitigate this (Leng et al., 2023). SDCD reports a similar precision–recall trade-off under excessive $x$23, and also finds that overly coarse shuffling weakens the structural contrast signal (Xia et al., 7 Jan 2026). CID warns that overly large $x$24 can produce degenerate or incoherent outputs and that the quality of insights depends on the quality of the perturbation $x$25 (Yona et al., 2023).

The choice of negative view is itself a source of brittleness. In the contextual QA method, if the irrelevant passage $x$26 accidentally contains relevant information, subtraction can harm performance; the paper therefore finds “most distant” negatives preferable to random ones (Zhao et al., 2024). SCICON reports that when the figure adds little beyond the text prior, subtracting the text-only branch can remove useful information and flip correct predictions to errors (Roh et al., 30 Mar 2026). The LLM-as-a-judge method likewise depends on selecting a same-family assistant and tuning $x$27 and assistant temperature $x$28; poorly tuned assistants can over-impose family bias rather than cancel it (Fujinuma, 21 Oct 2025).

Some limitations are architecture- or domain-specific. SDCD works best when the visual projector preserves patch-level independence, with gains smaller or mixed for Resampler-based projectors (Xia et al., 7 Jan 2026). SCICON is evaluated only in MCQA over scientific figures, and its extension to open-ended multimodal reasoning is described as unverified (Roh et al., 30 Mar 2026). The score-range-bias study is limited to English summarization tasks and models up to $x$29B parameters (Fujinuma, 21 Oct 2025).

Compute remains a recurring cost. Dual-view methods typically require two forward passes per step, and the tri-distribution QA method requires three. Anti-LM is an exception because it computes its bias term once per source sentence (Sia et al., 2023). SCD shows that contrastive subtraction can be made compatible with acceleration, but its benefits depend on acceptance rates, model cost ratio $x$30, and serving infrastructure (Yuan et al., 2023).

The open directions are correspondingly diverse. VCD proposes richer distortions such as object-level masking, region blurring, color jitter, or occlusions, and extension to multi-image, multi-turn dialogue, and spatial-reasoning tasks (Leng et al., 2023). CID suggests multi-contrast inputs, adaptive $x$31, and span-level contrast (Yona et al., 2023). The contextual QA method suggests multiple negatives and alternative aggregation rules (Zhao et al., 2024). SCICON points toward adaptive weighting based on prior alignment or confidence (Roh et al., 30 Mar 2026). Taken together, these proposals indicate that the future of Bias-Subtracted Contrastive Decoding lies less in a single canonical formula than in better design of bias proxies, adaptive weighting schemes, and task-specific negative views.