SafeCoDe: Safety-Aware Contrastive Decoding

• SafeCoDe is a decoding-time safety framework that integrates visual and textual modalities to make context-sensitive refusal decisions.
• It employs contrastive decoding by comparing base logits with those from a neutralized image, dynamically tuning responses based on scene safety.
• Experimental results show improved safety accuracy and reduced oversensitivity while maintaining near-original utility in multimodal applications.

SafeCoDe refers to Safety-aware Contrastive Decoding, a decoding-time safety framework for multimodal LLMs (MLLMs) designed to improve context-sensitive refusal behaviors while preserving the utility of generated responses. SafeCoDe is distinct from earlier code-focused approaches such as AutoSafeCoder and SecCoder but shares with these systems an emphasis on robust, generalizable safety solutions. Its methodology is characterized by integration of visual and textual modalities to balance the competing demands of caution and helpfulness in MLLMs, specifically addressing limitations such as oversensitivity and undersensitivity in refusal decisions (Liu et al., 23 Sep 2025).

1. Motivation and Problem Space

The deployment of MLLMs in real-world applications necessitates mechanisms for reliable context-sensitive safety judgments. Traditional text-only or naive multimodal approaches typically default to rigid heuristics, resulting in two main failure modes:

• Undersensitivity: The model overlooks visually implied risks and provides unsafe responses to hazardous scenarios latent in images (e.g., dangerous physical activities in unsafe settings).
• Oversensitivity: The model unjustifiably refuses benign user queries due to overreactive safety triggers tied to non-threatening images or ambiguous prompts.

Existing prompting or decoding schemes inadequately balance these risks. SafeCoDe aims to achieve both (i) visually grounded safety judgments and (ii) global scene-aware adaptations within an efficient, inference-time-only framework adaptable to arbitrary open-source MLLMs (Liu et al., 23 Sep 2025).

2. SafeCoDe Architecture and Workflow

The SafeCoDe framework consists of two sequential mechanisms during decoding:

2.1 Contrastive Decoding Initialization

For an input image $v$, user prompt $x$, and previously generated token sequence $y_{<t}$, SafeCoDe introduces a contrastive comparison:

• Construct $\tilde v$ as a "neutralized" version of $v$, obtained by adding Gaussian noise that preserves color and texture but disrupts semantic content.
• Compute base logits $\mathbf z_t(v,x,y_{<t})$ and contrastive logits $\mathbf z_t(\tilde v,x,y_{<t})$ at decoding step $t$.
• Form contrastive logits:

$$\mathbf z_t^{\rm cd} = \mathbf z_t(v,x,y_{<t}) - \alpha \cdot \mathbf z_t(\tilde v,x,y_{<t})$$

Here, $\alpha \in [0,1]$ modulates the noise subtraction. The softmax-normalized contrastive logits, $x$0, serve as the decoding distribution in early steps, amplifying response tokens that are highly sensitive to authentic visual details.

2.2 Global-Aware Token Modulation

• Generate a scene-level caption $x$1 for the image.
• Obtain a binary scene safety verdict $x$2 using an auxiliary MLLM-Judge, e.g., GPT-4V, which evaluates $x$3.
• For each decoding step $x$4, modulate the base logits $x$5 for a predefined refusal token set $x$6:

$x$7

The hyperparameters $x$8 control the encouragement or suppression of typical refusal tokens, thus tuning the strength of help/refusal responses according to safety context.

• Revert to ordinary sampling for $x$9, since refusal/help intent is typically established in early tokens.

The combination of mechanisms ensures that refusal and compliance behaviors are dynamically and contextually adapted using both local token-level signals and holistic scene-level verdicts (Liu et al., 23 Sep 2025).

3. Algorithmic Description

SafeCoDe’s end-to-end decoding process is detailed by the following algorithm:

1. Compute image caption $y_{<t}$0.
2. Evaluate safety verdict $y_{<t}$1 with the MLLM-Judge on $y_{<t}$2.
3. For each token position $y_{<t}$3:
   • If $y_{<t}$4:
     • Compute contrastive logits and obtain preliminary logit vector.
     • Apply token modulation over the refusal token set based on $y_{<t}$5.
     • Sample next token from modulated distribution.
   • Else:
     • Use ordinary logits for decoding.

This design is model-agnostic—requiring only white-box access to the MLLM’s logits—and can wrap any open-source MLLM at inference without retraining (Liu et al., 23 Sep 2025).

4. Experimental Setup and Benchmarks

SafeCoDe was evaluated across multiple MLLMs—LLaVA-1.6-7B, Qwen2.5-VL-7B-Instruct, InstructionBlip-7B, Idefics-9B-Instruct—against several baselines including vanilla decoding, CoT+Agg, Self-Examination, DPP, AdaShield, and others. Contextual safety and utility were measured on:

• MSSBench (undersensitivity): accuracy on cases where visual context changes safety ground truth
• MOSSBench (oversensitivity): benign prompt rejection rates across visual categories
• General Safety: MM-SafetyBench, FigStep, Hades—reporting attack success rate (ASR)
• Utility: MMMU, MIA-Bench, MathVista, MMVet

The table summarizes principal experimental results for SafeCoDe:

Model	MSSBench (↑)	MOSSBench (↓)
LLaVA-1.6-7B	+23.32 pp	–1.33 pp
Qwen2.5-VL-7B	+9.94 pp	–1.35 pp
InstructionBlip-7B	+16.48 pp	–2.00 pp
Idefics-9B	+14.87 pp	–9.66 pp

On MSSBench, these denote percentage point improvements in accuracy for safe/unsafe classification; on MOSSBench, reductions in rejection rates for benign prompts (Liu et al., 23 Sep 2025).

SafeCoDe reduces ASR to near zero on robust general safety/jailbreak benchmarks and maintains utility metrics within 1–2 percentage points of the base MLLMs.

5. Analysis, Practical Impact, and Limitations

SafeCoDe enables fine-grained trade-off tuning between safety and helpfulness via hyperparameters $y_{<t}$6, $y_{<t}$7, $y_{<t}$8, and $y_{<t}$9. In practice, 2–5 early decoding steps suffice for effective modulation. The system’s dependence on access to model logits limits its use to open-source or white-box MLLMs; future directions include black-box surrogate approximations.

Current scope is limited to amplifying or suppressing predefined refusal/compliance tokens. Expanding to “soft refusals” or richer safe-completion generators—e.g., generating partial advice while avoiding explicit unsafe content—is identified as a promising enhancement. The need for efficient, reliable scene-level safety judges highlights an open problem in multimodal safety verification.

SafeCoDe’s inference-time, model-agnostic design positions it as a robust and practical approach to balancing context-aware refusal and utility in safety-critical MLLM applications (Liu et al., 23 Sep 2025).