Contrastive Decoding Diffing (CDD)

• CDD is a decoding paradigm that employs token-level logit differencing between an expert model and a negative branch to steer generation outputs.
• It adapts to various applications by leveraging negative signals from model weaknesses, input corruption, or adverse prompts to suppress undesirable outputs.
• CDD has demonstrated practical gains in text coherence, code synthesis pass rates, and speech recognition accuracy while requiring careful tuning of negative signal parameters.

Contrastive Decoding Diffing (CDD) denotes a family of inference-time decoding procedures that alter next-token selection by explicitly contrasting a preferred distribution or logit stream against a negative, weaker, corrupted, or counterfactual alternative. In the literature, the label is not fully standardized: some papers use “Contrastive Decoding” for expert–amateur differencing, some map “CDD” informally onto task-specific variants such as unlearning or layer/input differencing, and one paper uses “Contrastive Decoding Diffing” as the formal method name for grey-box recovery of finetuning priors. Across these usages, the common mechanism is token-level differencing in log-probability, logit, probability, or attention space, performed entirely at test time or decoding time, usually without retraining (Li et al., 2022, Suriyakumar et al., 12 Jun 2025, Ahn et al., 6 Mar 2026, Brzozowski et al., 25 May 2026).

1. Terminology and conceptual scope

CDD is best understood as an umbrella description rather than a single canonical algorithm. In Li et al.’s open-ended generation formulation, the contrast is between a large “expert” LLM and a smaller “amateur” model, with decoding guided by the difference in their token likelihoods (Li et al., 2022). Subsequent work preserved the differencing principle while changing the source of the negative signal: forget-tuned versus retain-tuned auxiliaries for machine unlearning, clean versus acoustically degraded inputs for long-form ASR, relevant versus irrelevant context passages for contextual grounding, positive versus negative polarity prompts for behavior control, and finetuned versus base models for model auditing (Suriyakumar et al., 12 Jun 2025, Ahn et al., 6 Mar 2026, Zhao et al., 2024, Bi et al., 24 Feb 2026, Brzozowski et al., 25 May 2026).

Several papers explicitly note that “CDD” is informal in their setting. The DCD paper states that it does not define a method explicitly called “Contrastive Decoding Diffing,” and uses the phrase mainly to describe hidden-state or logit-diffing approaches such as DoLa (Phan et al., 2024). The confidence-driven reasoning paper likewise observes that the official name is CCD, while “CDD” is a reasonable informal label because the method performs subtraction of a confused reference distribution at selected positions (Tang et al., 20 Feb 2026). By contrast, “Reading the Finetuning Prior: Verbatim Content Recovery via Contrastive Decoding Diffing” formalizes CDD as a grey-box, output-level model-diffing method with no weight access, no layer selection, and no per-model tuning (Brzozowski et al., 25 May 2026).

Usage in the literature	Diffed objects	Representative scoring rule
Contrastive Decoding	expert LM vs amateur LM	$\log p_E - \log p_A$
Whisper-CD	clean-audio logits vs perturbed-audio negatives	$$(1+\alpha)z^{(c)}-\alpha\,\log\!\big(\frac{1}{K}\sum_i e^{z^{(i)}}\big)$$
UCD / CDD-style unlearning	reference logits vs forget/retain auxiliary diff	$\log P \leftarrow \log P - \alpha\Delta_A$
PromptCD	positive-prompt vs negative-prompt token distributions	$\log \mathbb{P}^{(P)}-\gamma\log \mathbb{P}^{(N)}$
Finetuning-prior CDD	finetuned model vs base model	$$(1+\beta)\log p_{\text{ft}}-\beta\log p_{\text{base}}$$

This terminological variation is not merely cosmetic. It indicates that the core idea has migrated from model-scale contrast in open-ended generation to a broader class of inference-time differencing schemes spanning language, speech, code, and multimodal systems.

2. Core mathematical structure

The simplest CDD objective is sequence-level contrastive decoding. Given a prompt $x$ and continuation $y=y_{1:T}$, classic CD defines

$$L_{\mathrm{CD}}(y\mid x)=\log p_E(y\mid x)-\log p_A(y\mid x),$$

which factorizes tokenwise and yields a per-step score of the form $$\log p_E(y_t\mid x,y_{<t})-\log p_A(y_t\mid x,y_{<t})$$ (Li et al., 2022). In practice, many implementations refactor the same idea directly in logit space, using

$s=(1+\beta)\,s_e-\beta\,s_a,$

or task-specific notational variants such as $$(1+\alpha)z^{(c)}-\alpha\,\log\!\big(\frac{1}{K}\sum_i e^{z^{(i)}}\big)$$0 for a single negative branch (O'Brien et al., 2023, Phan et al., 2024, Ahn et al., 6 Mar 2026).

A second recurring component is a plausibility or head constraint. Li et al. define an adaptive plausibility set

$$(1+\alpha)z^{(c)}-\alpha\,\log\!\big(\frac{1}{K}\sum_i e^{z^{(i)}}\big)$$1

and only score candidates inside that set (Li et al., 2022). Closely related filters reappear in later work: PromptCD uses an Adaptive Plausibility Constraint derived from the positive prompt, MACD uses a plausibility head from the base video-conditioned view, and USCD restricts subtraction to a rationality-filtered token set (Bi et al., 24 Feb 2026, Xiao et al., 2 Feb 2026, Wang et al., 2024). The role of the constraint is consistent across papers: contrast is useful only when it reorders plausible candidates rather than elevating off-manifold tokens.

A third pattern is aggregation over multiple negatives. Whisper-CD makes this explicit by replacing a single negative branch with a log-sum-exp aggregate over three perturbed-audio branches,

$$(1+\alpha)z^{(c)}-\alpha\,\log\!\big(\frac{1}{K}\sum_i e^{z^{(i)}}\big)$$2

followed by

$$(1+\alpha)z^{(c)}-\alpha\,\log\!\big(\frac{1}{K}\sum_i e^{z^{(i)}}\big)$$3

This converts differencing from a pairwise operation into a soft maximum over several failure-revealing perturbations (Ahn et al., 6 Mar 2026). A plausible implication is that CDD is not restricted to binary subtraction; it can be generalized to structured negative ensembles as long as the decoder ultimately receives a single adjusted score per token.

3. Construction of the negative or counterfactual branch

The quality of a CDD system depends heavily on how the negative signal is defined. The original expert–amateur setup relies on scale asymmetry: the weaker model amplifies failure modes such as repetition, incoherence, or shallow continuations, making those modes subtractable at decoding time (Li et al., 2022, O'Brien et al., 2023). Later work replaced model weakness with input corruption, prompt polarity, or counterfactual evidence suppression.

In long-form ASR, Whisper-CD computes negative logits from three acoustically motivated perturbations: additive Gaussian noise at $$(1+\alpha)z^{(c)}-\alpha\,\log\!\big(\frac{1}{K}\sum_i e^{z^{(i)}}\big)$$4, an all-zero silence signal, and a $$(1+\alpha)z^{(c)}-\alpha\,\log\!\big(\frac{1}{K}\sum_i e^{z^{(i)}}\big)$$5 left temporal shift with zero-padding. All branches use the same Whisper model and the same text prefix; only the audio differs (Ahn et al., 6 Mar 2026). In contextual grounding for QA, the negative branch is an irrelevant or adversarial passage $$(1+\alpha)z^{(c)}-\alpha\,\log\!\big(\frac{1}{K}\sum_i e^{z^{(i)}}\big)$$6, while the positive branch is a retrieved relevant passage $$(1+\alpha)z^{(c)}-\alpha\,\log\!\big(\frac{1}{K}\sum_i e^{z^{(i)}}\big)$$7; decoding uses $$(1+\alpha)z^{(c)}-\alpha\,\log\!\big(\frac{1}{K}\sum_i e^{z^{(i)}}\big)$$8 to balance parametric and contextual knowledge (Zhao et al., 2024). In PromptCD, the two branches are induced by positive and negative polarity prompts, and decoding remains synchronized by appending each chosen token to both prompt-conditioned histories (Bi et al., 24 Feb 2026).

Other variants construct negatives by deliberately damaging the model’s access to task-relevant structure. USCD removes input–output examples from a standard few-shot code prompt to create a “lame prompt,” then selectively subtracts its distribution only when the standard-prompt distribution is sufficiently uncertain (Wang et al., 2024). MACD uses the video model’s own gradients to optimize object- and frame-level masks, generating a targeted counterfactual video $$(1+\alpha)z^{(c)}-\alpha\,\log\!\big(\frac{1}{K}\sum_i e^{z^{(i)}}\big)$$9 rather than a random perturbation (Xiao et al., 2 Feb 2026). Confidence-Driven Contrastive Decoding for reasoning replaces previously generated high-confidence tokens with minimal placeholders in a second KV cache, creating a confused reference distribution used only at low-confidence steps (Tang et al., 20 Feb 2026).

These constructions reveal a shared design principle: the negative branch should preserve enough structure to remain comparable to the main branch, while removing, weakening, or misaligning precisely the evidence whose absence should expose hallucination, memorization, or shallow heuristics.

4. Major application domains

CDD first gained visibility in open-ended text generation. Li et al. reported that classic CD improves MAUVE and coherence relative to standard sampling and search baselines across Wikipedia, news, and story domains, while the later reasoning study showed that LLaMA-65B with contrastive decoding reaches 57.7 on GSM8K and 88.0 on HellaSwag, exceeding several named baselines in the reported comparisons (Li et al., 2022, O'Brien et al., 2023). At the same time, the empirical comparison of CD and Contrastive Search found that human annotators preferred CS over CD across all three domains, despite CD’s higher MAUVE, which exposed an important evaluation mismatch (Su et al., 2022).

In code generation, USCD adapts the differencing idea to one-pass program synthesis. The paper reports an average pass@$\log P \leftarrow \log P - \alpha\Delta_A$0 scores increase of 16.59\% across HumanEval, MBPP, and MultiPL-E settings, with gains such as +10.94\% for InCoder-6B on HumanEval and +19.25\% for InCoder-6B on MultiPL-E (Wang et al., 2024). In speech recognition, Whisper-CD applies multi-negative differencing to long-form ASR and reports five-benchmark improvements including Whisper Large-v3-Turbo on CORAAL from 38.75 to 14.43 WER and throughput of 147.0 tokens/s versus 99.0 for beam search (Ahn et al., 6 Mar 2026).

Behavior control and grounding have produced additional task-specific generalizations. PromptCD reports substantial improvements on the “3H” alignment objectives for LLMs and improved VQA performance for VLMs, including LLaMA2-7B-chat on NQ with ConR 75.99\% versus 43.35\% for vanilla and LLaVA-1.5-7B on TextVQA from 47.8\% to 58.2\% (Bi et al., 24 Feb 2026). MACD applies model-aware counterfactual differencing to Video-LLMs and reports, for example, Qwen3-VL-2B on EventHallusion F1 from 0.683 to 0.847 and on MVBench accuracy from 0.547 to 0.773 (Xiao et al., 2 Feb 2026).

Two further branches extend CDD beyond generation quality. UCD performs inference-time machine unlearning by subtracting the token preference difference between a forget-tuned and a retain-tuned auxiliary model; it achieves indistinguishability from retraining on TOFU 5\% and scales to Llama2-70B where NPO/NPO+RT were OOM (Suriyakumar et al., 12 Jun 2025). The finetuning-prior auditing paper uses CDD to recover implanted facts from narrowly finetuned models, achieving mean score $\log P \leftarrow \log P - \alpha\Delta_A$1 on 16/20 pairs under a strict verbatim rubric and running $\log P \leftarrow \log P - \alpha\Delta_A$2 faster than ADL while requiring less access (Brzozowski et al., 25 May 2026).

Domain	Representative reported result	Paper
Open-ended generation	OPT-13B CD on wikinews: MAUVE 0.94, coherence 0.69	(Li et al., 2022)
Reasoning	LLaMA-65B + CDD on GSM8K: 57.7	(O'Brien et al., 2023)
Long-form ASR	CORAAL WER 38.75 $\log P \leftarrow \log P - \alpha\Delta_A$3 14.43; 147.0 vs 99.0 tokens/s	(Ahn et al., 6 Mar 2026)
Code generation	Average pass@$\log P \leftarrow \log P - \alpha\Delta_A$4 scores increase of 16.59\%	(Wang et al., 2024)
Behavior control / VQA	TextVQA 47.8\% $\log P \leftarrow \log P - \alpha\Delta_A$5 58.2\% on LLaVA-1.5-7B	(Bi et al., 24 Feb 2026)
Finetuning-prior auditing	mean score $\log P \leftarrow \log P - \alpha\Delta_A$6 on 16/20 pairs; $\log P \leftarrow \log P - \alpha\Delta_A$7 faster than ADL	(Brzozowski et al., 25 May 2026)

5. Empirical behavior, failure modes, and controversies

Despite strong empirical results, CDD does not have a uniform failure profile. The APD paper gives the clearest formal critique of classic expert–amateur CD: linear logit extrapolation can cause “obvious blindness,” in which tokens already assigned high probability by the amateur model are downweighted even when they are the most obvious correct answers (Chang et al., 2024). This diagnosis is task-dependent. The reasoning paper likewise reports that CDD improves abstract reasoning and reduces copying and missing-step errors, but may slightly hurt pure arithmetic evaluation and factual recall, indicating that subtraction amplifies some expert advantages while not inventing absent capabilities (O'Brien et al., 2023).

Several papers stress that the negative branch must be conservative and well matched to the task. Whisper-CD reports that no single acoustic negative wins universally: silence helps CORAAL but hurts TED-LIUM, and the authors explicitly note that silence negative can hurt clean data, motivating multi-negative aggregation and conservative $\log P \leftarrow \log P - \alpha\Delta_A$8 (Ahn et al., 6 Mar 2026). UCD depends on clean auxiliary models; when only approximate auxiliaries are available, the paper recommends the one-sided UCS variant to prevent noisy negative differences from boosting undesirable tokens (Suriyakumar et al., 12 Jun 2025). PromptCD notes that gains diminish when the negative prompt resembles the positive prompt, and that excessive $\log P \leftarrow \log P - \alpha\Delta_A$9 can degrade fluency or over-suppress useful priors (Bi et al., 24 Feb 2026).

The necessity of plausibility constraints is another recurrent theme. Li et al. report that removing the plausibility set causes severe fluency issues, with MAUVE collapsing to approximately 0.01 and perplexity exploding (Li et al., 2022). Similar warnings appear in context-grounding, behavior-control, and finetuning-prior extraction settings, where head masking prevents the subtraction term from promoting tokens that the preferred branch itself considers implausible (Zhao et al., 2024, Bi et al., 24 Feb 2026, Brzozowski et al., 25 May 2026).

A distinct controversy concerns evaluation. The empirical study comparing Contrastive Search and Contrastive Decoding found that MAUVE favored CD while human annotators preferred CS by substantial margins across Wikinews, Wikitext-103, and BookCorpus, leading the authors to argue that MAUVE does not accurately reflect human preferences for open-ended generation (Su et al., 2022). This suggests that CDD improvements are often metric-contingent and should be interpreted alongside task-specific failure analysis rather than through a single global score.

6. Relation to adjacent methods and broader significance

CDD overlaps with, but is not identical to, several neighboring decoding paradigms. Contrastive Search is single-model and penalizes hidden-state similarity rather than subtracting an explicit negative distribution (Su et al., 2022). DoLa and related hidden-state/logit-diffing methods contrast internal layers instead of external branches; the DCD paper presents this family as a form of hidden-state diffing and positions DCD as a prompt-based alternative that avoids layer instrumentation (Phan et al., 2024). DExperts and classifier-free-guidance-like methods share the linear-combination form, but their negatives are typically anti-expert models or unconditional branches rather than the task-conditioned counterfactuals used in PromptCD, Whisper-CD, MACD, or finetuning-prior CDD (Phan et al., 2024, Bi et al., 24 Feb 2026, Brzozowski et al., 25 May 2026).

Input-contrast variants also broaden the concept beyond model pairs. Contrastive Input Decoding constructs generations likely under one input but unlikely under a contrastive version of that input, chiefly to surface subtle biases that standard decoding may hide (Yona et al., 2023). Contextual-understanding CDD contrasts relevant and irrelevant passages for the same model, effectively moving the subtraction target from model scale to evidence source (Zhao et al., 2024). The finetuning-prior auditing paper pushes this logic further: the contrasted objects are not two prompts or two model scales, but the output distributions of a base model and its narrowly finetuned derivative, in raw next-token-prediction mode (Brzozowski et al., 25 May 2026).

This suggests that CDD is more usefully classified as a decoding pattern than as a single named algorithm. Its defining operation is the inference-time extraction of a differential signal from two or more comparable conditions, followed by token selection under a constrained adjusted score. What changes across papers is the semantics of the contrast: model competence, retained versus forgotten content, contextual evidence versus distraction, behavior-aligned versus behavior-opposed prompting, clean versus degraded acoustics, or base versus finetuned priors. The resulting research trajectory has turned contrastive differencing from a niche text-generation decoder into a general-purpose mechanism for hallucination suppression, robustness improvement, behavior control, unlearning, and model transparency (Li et al., 2022, Suriyakumar et al., 12 Jun 2025, Ahn et al., 6 Mar 2026, Brzozowski et al., 25 May 2026).