SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models

Abstract: Multimodal LLMs (MLLMs) are gaining increasing attention. Due to the heterogeneity of their input features, they face significant challenges in terms of jailbreak defenses. Current defense methods rely on costly fine-tuning or inefficient post-hoc interventions, limiting their ability to address novel attacks and involving performance trade-offs. To address the above issues, we explore the inherent safety capabilities within MLLMs and quantify their intrinsic ability to discern harmfulness at decoding stage. We observe that 1) MLLMs can distinguish the harmful and harmless inputs during decoding process, 2) Image-based attacks are more stealthy. Based on these insights, we introduce SafeSteer, a decoding-level defense mechanism for MLLMs. Specifically, it includes a Decoding-Probe, a lightweight probe for detecting and correcting harmful output during decoding, which iteratively steers the decoding process toward safety. Furthermore, a modal semantic alignment vector is integrated to transfer the strong textual safety alignment to the vision modality. Experiments on multiple MLLMs demonstrate that SafeSterr can improve MLLMs' safety by up to 33.40\% without fine-tuning. Notably, it can maintain the effectiveness of MLLMs, ensuring a balance between their helpfulness and harmlessness.

• The paper introduces a novel decoding-level defense that uses a lightweight Decoding-Probe to rerank candidate tokens based on safety scores, reducing harmful outputs.
• It integrates a Modal Semantic Alignment Vector to adaptively counter both text-based and image-based adversarial attacks while preserving semantic fidelity.
• Empirical evaluations demonstrate up to a 33.40% improvement in safety metrics with minimal computational overhead, promising safer deployments in high-risk applications.

SafeSteer: Decoding-Level Defense for Multimodal LLMs

Motivation and Background

Multimodal LLMs (MLLMs), capable of processing heterogenous modalities such as text, images, and audio, present substantial safety challenges due to their expansive attack surface and fragile cross-modal alignment mechanisms. Existing defense paradigms—primarily fine-tuning and post-hoc input/output interventions—demonstrate limitations in generalization and computational efficiency. Fine-tuning is costly and brittle, frequently failing to mitigate novel and stealthy attacks, especially those leveraging modality-specific vulnerabilities. Input/output rewriting often produces over-corrected or semantically degraded responses, impinging on model helpfulness.

A critical insight underpinning this work is that all generative outputs, regardless of input modality, are rendered by the MLLM's text decoder. Thus, decoder-level interventions, leveraging the inherent discriminative capacity of MLLMs during generation, could provide more robust, efficient safety mechanisms.

Figure 1: Comparative illustration of input/output intervention, fine-tuning, and SafeSteer’s stepwise correction paradigm during decoding.

Inherent Discriminability and Attack Analysis

The authors conducted extensive visualization analyses of hidden states (prefill and decoding stages), applying PCA and t-SNE to expose latent separability between harmful and benign queries. Results reveal a distinct spatial margin at the prefill stage, evidencing the intrinsic ability of MLLMs to differentiate harmfulness. Stealthy image-based attacks induce a weaker harmfulness representation in the decoder space, facilitating attack success by shifting semantics to the visual modality—a direct consequence of imperfect multimodal alignment.

Figure 2: Probe performance visualized at prefill and decoding stages; image-based attacks manifest closer to benign clusters.

SafeSteer: Methodology

SafeSteer operates directly at the decoding level, introducing two principal components: Decoding-Probe and Modal Semantic Alignment Vector (MSAV).

Decoding-Probe

The Decoding-Probe is a lightweight linear classifier (logistic regression) trained on hidden states, assigning harmfulness scores to candidate tokens during generation. Unlike standard logit-based sampling, SafeSteer reranks the top-k candidate tokens by their probe-derived safety scores, iteratively steering token selection toward benign content while maintaining fluency.

Modal Semantic Alignment Vector

To address visual modality vulnerabilities, SafeSteer computes a MSAV via centroid shifts in decoder hidden space, derived from text-based (CB) and image-based (SD) attack samples. When classifying an input as harmful, the MSAV is adaptively added to the prefill stage hidden state, transferring strong textual safety alignment to vision and mitigating stealthy image attacks.

Figure 3: Overview of SafeSteer’s architecture, depicting extraction of MSAV, Decoding-Probe training, and intervention in decoding.

Empirical Evaluation

SafeSteer was evaluated across multiple MLLMs (Qwen2.5-VL, Qwen3-VL, LLaVA-1.5-7b) and safety benchmarks (MM-SafetyBench, FigStep, VL-Guard). Metrics included Attack Success Rate (ASR), Refusal Rate (RR), and Accuracy (Acc). SafeSteer achieved up to 33.40% improvement in safety (lower ASR) without trading off effectiveness of benign task completion (stable or improved Accuracy, lower RR compared to aggressive rewriting baselines). Robust generalization was observed even on out-of-domain attack sets.

Ablation studies demonstrated that the Decoding-Probe is particularly effective against text-based harmful inputs, while MSAV is crucial for image-based attacks. Removal of either component significantly degraded defense performance.

Figure 4: Training loss convergence for Decoding-Probe, illustrating rapid and stable optimization across models.

Figure 5: Case study—SafeSteer progressively corrects harmful output in MM-SafetyBench, guiding response to safe refusal.

Efficiency analysis confirmed SafeSteer’s negligible computational overhead: probe fitting required only 0.03 hours on Qwen2.5-VL, with inference throughput nearly identical to vanilla MLLMs.

Design Advantages and Comparative Discussion

SafeSteer’s plug-and-play architecture leverages the MLLM’s endogenous safety features, functioning as a decoding-time, single-pass intervention without modifying core model parameters or requiring expensive retraining. By focusing on token-level discrimination at the point of generation, it avoids semantic distortion and maintains fluency. The adaptive MSAV offers targeted cross-modal resilience, exploiting latent spatial shifts to surface otherwise masked harmful semantics in the visual channel.

Notably, SafeSteer diverges from input/output-level defenses and external detector-based approaches, fundamentally aligning the defense mechanism with internal MLLM representations and operational flow.

Figure 6: Hyperparameter study—Effect of varying step and k settings on performance.

Figure 7: Probe performance across decoding layers for Qwen2.5-VL and Qwen3-VL.

Figure 8: Probe performance visualized at different steps during decoding; harmful/harmless outputs become less separable at later steps.

Practical and Theoretical Implications

Practically, SafeSteer facilitates safer deployment of multimodal foundation models in high-risk applications (e.g., visual assistant APIs, autonomous systems) by reliably filtering cross-modal adversarial payloads within the generative process. Theoretically, the work robustly quantifies and leverages latent safety alignment in MLLMs, empirically validating that decoder-layer interventions can surpass brittle surface-level defenses. The use of spatial centroid-based MSAV for cross-modal alignment transfer establishes a formal framework for exploiting internal representations against novel, stealthy attacks.

Future developments may extend adaptive, layer-wise steering, tackle multimodal adversarial training minimally invasively, and generalize MSAV to additional modalities (audio, structured data). Integrating SafeSteer with reinforcement learning safety tuning or auto-remediation frameworks may yield further robustness.

Conclusion

SafeSteer establishes a decoding-level defense paradigm for MLLMs, leveraging internal discriminative capability and adaptive semantic alignment interventions to achieve substantial safety improvements with minimal computational overhead. Empirical results demonstrate superior generalization and efficiency compared to existing approaches, validating the efficacy of probing-based steering for multimodal safety without degrading utility.