Safety-aware Contrastive Decoding (SafeCoDe)

• Safety-aware Contrastive Decoding (SafeCoDe) is a paradigm that uses contrasting token distributions to adjust model behavior without retraining.
• It implements methods like AdaCD, ROSE, and ACD to modulate refusal tokens by comparing base and safety-conditioned outputs.
• Adaptive and context-aware strategies, including multimodal extensions, reduce over-refusal while maintaining high security against unsafe prompts.

Searching arXiv for the cited papers and closely related work to ground the article. Searching arXiv for “Safety-aware Contrastive Decoding”, “Adaptive Contrastive Decoding over-refusal”, “Adversarial Contrastive Decoding”, and “ROSE reverse prompt contrastive decoding”. Safety-aware Contrastive Decoding (SafeCoDe) denotes a family of inference-time decoding strategies in which next-token selection is steered by contrasting a base generation distribution with a safety-conditioned, safety-opposed, or context-neutralized distribution, thereby increasing or decreasing the probability of refusal-related tokens without retraining the underlying model. In the AdaCD formulation, SafeCoDe is described as the broader principle of using contrastive signals tied to safety to modulate decoding in a safety-aware manner, with Adaptive Contrastive Decoding (AdaCD) as a text-only instantiation (Qi et al., 18 Apr 2026). In multimodal settings, the term is used explicitly for a two-stage framework that contrasts real and Gaussian-noised images and then modulates refusal tokens according to a global safety verdict (Liu et al., 23 Sep 2025). Closely related methods such as ROSE and Adversarial Contrastive Decoding (ACD) do not use the name “SafeCoDe” explicitly, but they implement the same general logic of constructing a negative or adversarial decoding signal and subtracting it during inference (Zhong et al., 2024, Zhao et al., 2024).

1. Definition and conceptual scope

SafeCoDe addresses a recurring limitation of aligned LLMs and multimodal LLMs: safety behavior is often too coarse. In text-only systems, this appears as over-refusal, where harmless prompts that resemble malicious ones are declined. In multimodal systems, the analogous failure modes are oversensitivity, which produces unjustified refusals on benign image–text pairs, and undersensitivity, which misses visually grounded risks (Qi et al., 18 Apr 2026, Liu et al., 23 Sep 2025).

The central design principle is contrastive. A model is queried under two conditions that differ in safety emphasis, and the difference between their token distributions is used to alter decoding. Depending on the method, this alteration may suppress unsafe continuations, promote refusal continuations, or suppress refusal continuations when safety behavior is overly cautious. What distinguishes SafeCoDe from generic contrastive decoding is that the contrastive signal is explicitly safety-anchored rather than merely fluency- or quality-anchored (Qi et al., 18 Apr 2026).

Method	Contrastive signal	Control mechanism
ROSE	Positive prompt vs reverse prompt	Fixed logit subtraction
ACD	Safeguarding Prompt vs Adversarial Prompt	Learned opposite prompts via OPO
AdaCD	Base distribution vs Extreme safety prompt	Adaptive add/subtract via AGR and ACC
Multimodal SafeCoDe	Real image vs Gaussian-noised image, plus global verdict	Context-aware refusal modulation

This taxonomy also clarifies scope. SafeCoDe is not a single canonical algorithm. Rather, it is a decoding-time paradigm whose concrete instantiations differ in how they construct the safety contrast, whether the adjustment is fixed or adaptive, and whether the target behavior is unidirectional safety boosting or bidirectional safety/helpfulness balancing (Zhao et al., 2024, Liu et al., 23 Sep 2025).

2. Contrastive decoding formulations

Across the literature, SafeCoDe-like methods operate on per-step logits or probabilities. ROSE defines standard generation under a positive prompt $s$ as

$$p_\theta(y_t \mid s, x, y_{<t}) = \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t})],$$

and modifies decoding through

$$y_t \sim \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t}) - \alpha \cdot \mathrm{logit}_\theta(y_t \mid \hat{s}, x, y_{<t})],$$

where $\hat{s}$ is a reverse prompt and $\alpha$ is a nonnegative contrastive penalty (Zhong et al., 2024). ACD uses the same model-level structure but replaces manual prompts with optimized soft prompts, yielding

$\tilde{\ell}_t = \ell_{S,t} - \alpha \ell_{A,t},$

where $\ell_{S,t}$ comes from a Safeguarding Prompt and $\ell_{A,t}$ from an Adversarial Prompt learned by Opposite Prompt Optimization (Zhao et al., 2024).

AdaCD departs from fixed subtraction. Let $z_{\text{base}}(n)$ and $z_{\text{safe}}(n)$ denote logits at position $$p_\theta(y_t \mid s, x, y_{<t}) = \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t})],$$0 under a neutral prompt and under the extreme system prompt $$p_\theta(y_t \mid s, x, y_{<t}) = \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t})],$$1, with probabilities $$p_\theta(y_t \mid s, x, y_{<t}) = \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t})],$$2 and $$p_\theta(y_t \mid s, x, y_{<t}) = \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t})],$$3 given by softmax. AdaCD defines a refusal token distribution

$$p_\theta(y_t \mid s, x, y_{<t}) = \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t})],$$4

and then forms a safety-aware adjusted probability

$$p_\theta(y_t \mid s, x, y_{<t}) = \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t})],$$5

where $$p_\theta(y_t \mid s, x, y_{<t}) = \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t})],$$6 switches between addition and subtraction according to an agreement ratio and an adaptive confidence constraint (Qi et al., 18 Apr 2026). This adaptive sign switch is the defining extension over fixed contrastive policies.

In multimodal SafeCoDe, the contrastive signal is grounded in visual context rather than in alternative system prompts. For image $$p_\theta(y_t \mid s, x, y_{<t}) = \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t})],$$7 and Gaussian-noised image $$p_\theta(y_t \mid s, x, y_{<t}) = \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t})],$$8, the contrastive logits are

$$p_\theta(y_t \mid s, x, y_{<t}) = \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t})],$$9

after which refusal-token logits are boosted or suppressed using a scene-level verdict $$y_t \sim \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t}) - \alpha \cdot \mathrm{logit}_\theta(y_t \mid \hat{s}, x, y_{<t})],$$0 (Liu et al., 23 Sep 2025). The common structure across these methods is therefore token-level contrast, but the semantics of the negative branch differ: reverse prompt, adversarial prompt, extreme-safety prompt, or neutralized image.

3. AdaCD and the mitigation of over-refusal

AdaCD was introduced to address the over-refusal problem in safety-aligned LLMs, where harmless queries that superficially resemble malicious ones are declined. The method is motivated by two empirical observations. First, fixed decoding policies are asymmetric: methods that always suppress refusal tokens reduce over-refusal but can lower safety on malicious prompts, while methods that always boost refusal tokens improve safety but exacerbate over-refusal. Second, system prompts with different safety emphases materially change refusal behavior. Four prompts—Low, Medium, High, and Extreme (“Please refuse to answer me!”)—systematically modulate refusal ratios, and the Extreme prompt maximizes refusal behavior without yielding a purely refusal-only token distribution (Qi et al., 18 Apr 2026).

The key observation is token-local. In over-refusal cases, the next-token candidate list still contains non-refusal tokens, but the model systematically fails to choose them. The paper gives a concrete example: in an over-refusal case, Qwen3-8B’s top-5 first-token candidates contained “Sure” with non-zero probability (0.41%), yet the model selected “I” (60.80%), a typical refusal preamble (Qi et al., 18 Apr 2026). This motivates extracting refusal-correlated mass from the difference between extreme-safety and base distributions rather than defining a fixed refusal lexicon.

AdaCD restricts decoding to a plausibility-filtered candidate set

$$y_t \sim \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t}) - \alpha \cdot \mathrm{logit}_\theta(y_t \mid \hat{s}, x, y_{<t})],$$1

using $$y_t \sim \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t}) - \alpha \cdot \mathrm{logit}_\theta(y_t \mid \hat{s}, x, y_{<t})],$$2, lower than the conventional $$y_t \sim \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t}) - \alpha \cdot \mathrm{logit}_\theta(y_t \mid \hat{s}, x, y_{<t})],$$3, to preserve low-probability non-refusal tokens. It then identifies the top-1 token $$y_t \sim \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t}) - \alpha \cdot \mathrm{logit}_\theta(y_t \mid \hat{s}, x, y_{<t})],$$4 under the extreme-safety distribution, computes its rank $$y_t \sim \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t}) - \alpha \cdot \mathrm{logit}_\theta(y_t \mid \hat{s}, x, y_{<t})],$$5 in the base distribution, and defines the agreement ratio as $$y_t \sim \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t}) - \alpha \cdot \mathrm{logit}_\theta(y_t \mid \hat{s}, x, y_{<t})],$$6. With $$y_t \sim \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t}) - \alpha \cdot \mathrm{logit}_\theta(y_t \mid \hat{s}, x, y_{<t})],$$7, $$y_t \sim \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t}) - \alpha \cdot \mathrm{logit}_\theta(y_t \mid \hat{s}, x, y_{<t})],$$8, and threshold $$y_t \sim \mathrm{softmax}[\mathrm{logit}_\theta(y_t \mid s, x, y_{<t}) - \alpha \cdot \mathrm{logit}_\theta(y_t \mid \hat{s}, x, y_{<t})],$$9, AdaCD sets

$\hat{s}$0

and $\hat{s}$1 otherwise. High agreement and sufficiently strong base confidence cause the method to add $\hat{s}$2 and boost refusal; disagreement or weak base confidence causes it to subtract $\hat{s}$3 and suppress refusal (Qi et al., 18 Apr 2026).

The practical defaults are greedy decoding, $\hat{s}$4, and contrastive decoding only for the first $\hat{s}$5 positions, with $\hat{s}$6, $\hat{s}$7, and $\hat{s}$8. On five benchmark datasets, AdaCD reduces the refusal ratio for over-refusal queries by 10.35% on average relative to Default while increasing the refusal ratio for malicious queries by 0.13% on average. The reported model-wise over-refusal averages are Llama3: 32.57 → 16.62, Gemma2: 41.21 → 32.98, and Qwen3: 14.77 → 7.89; malicious-query averages are Qwen3: 99.44 → 99.68, Gemma2: 99.71 → 99.87, and Llama3: 99.28 → 99.10, which remains extremely high (Qi et al., 18 Apr 2026). AdaCD also attains the best average Just Eval usability score, 4.49, and achieves an Average Token Generation Time Ratio of approximately $\hat{s}$9 relative to Default.

A central empirical finding is that adaptivity is necessary. The paper reports that removing adaptivity and always using $\alpha$0 or always using $\alpha$1 breaks the safety/helpfulness balance; the agreement ratio and adaptive confidence constraint each contribute, and best performance occurs when both are active (Qi et al., 18 Apr 2026). This is the clearest text-only example of SafeCoDe as bidirectional control rather than monotone safety amplification.

4. Related text-only methods: ROSE and ACD

ROSE is a training-free contrastive decoding method for instruction-tuned LLMs that uses a carefully designed reverse system prompt to induce undesired or unsafe continuations and then suppresses them by direct logit subtraction. The paper studies Rand-Words, Opposite-Replace, and Manual-Reverse prompts, and reports that Manual-Reverse consistently yields the strongest contrast. ROSE evaluates six safety tasks and two general-purpose tasks, with reported improvements up to +13.98% safety score; one highlighted result is Alpaca-13B on HarmfulQA, which rises from 60.31 to 74.29, while DangerousQA rises from 71.50 to 85.00 and XSTest (unsafe) from 74.50 to 85.50 (Zhong et al., 2024). The method introduces no extra calibration terms beyond the scalar $\alpha$2, but it requires two forward passes per decoding step.

The ROSE paper does not mention “SafeCoDe” explicitly. Conceptually, however, it is a prompt-centric instance of safety-aware contrastive decoding: the model under the reverse prompt acts as a negative branch whose token probabilities are suppressed during generation (Zhong et al., 2024). Its main emphasis is unidirectional safety improvement rather than over-refusal mitigation. The paper notes that the main reported improvements are on unsafe prompts in XSTest and does not present explicit over-refusal rates on the safe subset.

ACD extends the same general pattern by learning the contrastive prompts rather than handcrafting them. Its Opposite Prompt Optimization procedure produces a Safeguarding Prompt and an Adversarial Prompt as soft prompts, keeping the target model weights frozen. The anchor dataset contains 200 instructions, split into 100 safe and 100 unsafe, and three sampled responses per instruction generated with manual prompts, producing 600 instruction–response pairs. ChatGPT labels the responses as “accepted” or “refused,” forming the subsets $\alpha$3, $\alpha$4, and $\alpha$5 used to optimize the prompt embeddings with Adam, batch size 16, learning rate $\alpha$6, for 5 epochs on a single NVIDIA A100 in less than 3 minutes per model (Zhao et al., 2024).

At inference, ACD uses

$\alpha$7

with $\alpha$8 by default and greedy decoding on the modified logits. On safety benchmarks, the average harmless rate across models and benchmarks improves from 0.714 under Base to 0.924 under ACD, described as +21.0% versus Base and +7.0% versus ID variants. On jailbreak defense for Mistral-7b-Instruct on AdvBench, the AIM prompt changes from HLR 0.21 → 0.75 and ASR 0.78 → 0.23, while Refusal Suppression changes from HLR 0.44 → 0.96 and ASR 0.41 → 0.02 (Zhao et al., 2024). The ACD paper also does not mention “SafeCoDe” explicitly, but it is described as a direct instance of safety-aware contrastive decoding in which the safe-aligned and adversarial directions are learned rather than manually specified.

Taken together, ROSE and ACD illustrate two precursor patterns within the SafeCoDe space. ROSE maximizes simplicity through prompt engineering and fixed subtraction; ACD replaces manual prompt selection with learned opposite prompts; AdaCD then adds adaptive sign switching to address the helpfulness–safety tension in over-refusal regimes.

5. Multimodal SafeCoDe and context-aware refusal

In multimodal LLMs, SafeCoDe is formulated explicitly as a context-aware safety framework. The motivating problem is that refusal behavior should depend jointly on textual intent and visual scene content, yet many MLLMs exhibit both oversensitivity and undersensitivity. The paper reports that replacing images with blanks barely changes performance on contextual safety benchmarks, indicating over-reliance on the textual modality and insufficient visual grounding (Liu et al., 23 Sep 2025).

The method has two stages. The first stage, contrastive decoding initialization, compares logits conditioned on the real image with logits conditioned on a Gaussian-noised image that preserves low-level statistics while neutralizing semantic content. This highlights tokens sensitive to meaningful visual evidence. The second stage, global-aware token modulation, obtains a scene-level safety verdict from an auxiliary MLLM judge that reasons jointly over the image, a caption, and the user query. The paper instantiates the judge with GPT-4o in the main experiments and notes that lighter judges such as Qwen2.5-3B-Instruct also work, with trade-offs (Liu et al., 23 Sep 2025).

Refusal-token modulation operates over a curated refusal token space $\alpha$9, with vocabulary indices $\tilde{\ell}_t = \ell_{S,t} - \alpha \ell_{A,t},$0. Given current logits $\tilde{\ell}_t = \ell_{S,t} - \alpha \ell_{A,t},$1 and verdict $\tilde{\ell}_t = \ell_{S,t} - \alpha \ell_{A,t},$2, SafeCoDe adjusts refusal-token logits by adding $\tilde{\ell}_t = \ell_{S,t} - \alpha \ell_{A,t},$3 when $\tilde{\ell}_t = \ell_{S,t} - \alpha \ell_{A,t},$4 and subtracting $\tilde{\ell}_t = \ell_{S,t} - \alpha \ell_{A,t},$5 when $\tilde{\ell}_t = \ell_{S,t} - \alpha \ell_{A,t},$6, leaving all other logits unchanged. The final distribution is

$\tilde{\ell}_t = \ell_{S,t} - \alpha \ell_{A,t},$7

Modulation is applied only during early steps, typically $\tilde{\ell}_t = \ell_{S,t} - \alpha \ell_{A,t},$8–$\tilde{\ell}_t = \ell_{S,t} - \alpha \ell_{A,t},$9, to seed the safety stance without over-regularizing the entire sequence (Liu et al., 23 Sep 2025).

The default hyperparameters reported in the appendix are Max Steps = 5 for LLaVA-1.6-7B and Qwen-VL-7B-Instruct, Max Steps = 2 for InstructionBlip-7B and Idefics-9B-Instruct, top_k = 20, $\ell_{S,t}$0, $\ell_{S,t}$1, and $\ell_{S,t}$2. Across four backbones, the paper reports consistent gains on MSSBench and lower benign rejection on MOSSBench. For example, LLaVA-1.6-7B improves from 50.76% to 74.08% on MSSBench overall average, while MOSSBench average rejection changes from 7.33% to 6.00%; Idefics-9B-Instruct improves from 50.76% to 65.63% on MSSBench overall average, while MOSSBench average rejection changes from 18.33% to 8.67% (Liu et al., 23 Sep 2025). The paper also states that SafeCoDe lowers attack success rate on MM-SafetyBench, Hades, and FigStep and drives FigStep ASR “down to nearly 0%” with LLaVA.

The multimodal formulation makes explicit a distinction that is only implicit in the text-only literature: safety-aware contrastive decoding need not merely suppress unsafe tokens. It can also suppress refusal tokens when the global context is benign. This bidirectional control is the multimodal analogue of AdaCD’s adaptive addition and subtraction of $\ell_{S,t}$3.

6. Limitations, misconceptions, and open technical questions

A common misconception is that safety-aware contrastive decoding simply means subtracting an unsafe distribution at every step. The literature does not support that reduction. ROSE and ACD do use fixed subtraction, but AdaCD reports that always adding or always subtracting the refusal distribution breaks the safety/helpfulness balance, and multimodal SafeCoDe uses both boosting and suppression depending on a global verdict (Qi et al., 18 Apr 2026, Liu et al., 23 Sep 2025). SafeCoDe is therefore better understood as safety-conditioned contrastive control rather than as a single subtraction rule.

A second misconception is that prompt engineering alone solves the problem. The reported methods rely on prompt choice, but their operative mechanism is access to token logits under multiple conditions and intervention inside the decoding loop. AdaCD requires computing base and extreme-safety distributions, AGR and ACC gates, and a plausibility-filtered candidate set; multimodal SafeCoDe requires two image-conditioned forward passes plus an auxiliary verdict; ROSE and ACD each require two prompt-conditioned branches during decoding (Qi et al., 18 Apr 2026, Zhao et al., 2024, Liu et al., 23 Sep 2025).

Several limitations recur. AdaCD evaluates only 8–9B models and notes that behavior on $\ell_{S,t}$4B models remains untested; its performance also depends on the quality of refusal distribution extraction from the Extreme prompt and can degrade when $\ell_{S,t}$5 is too large (Qi et al., 18 Apr 2026). ROSE and ACD double per-step computation because they require two forward passes, and ROSE notes that formal guarantees are absent (Zhong et al., 2024, Zhao et al., 2024). Multimodal SafeCoDe requires logit access, making fully black-box deployment difficult, and its effectiveness depends on the MLLM judge and on the heuristic early-step modulation window (Liu et al., 23 Sep 2025).

The open research direction suggested by these results is not merely stronger refusal. It is finer control over when refusal should be expressed, when it should be suppressed, and how strongly the model should condition on evidence from prompts, learned adversarial branches, or visual context. AdaCD explicitly recommends query-adaptive $\ell_{S,t}$6 and $\ell_{S,t}$7 as a possible improvement, while multimodal SafeCoDe identifies adaptive step-length tuning and more informative safe-completion behavior as future directions (Qi et al., 18 Apr 2026, Liu et al., 23 Sep 2025). In that sense, SafeCoDe design is increasingly moving from static safety alignment toward conditional, context-sensitive decoding policies.