Safe Semantics but Unsafe Interpretation (SSUI)

• SSUI is a cross-modal safety failure where individually safe inputs combine to produce unsafe global interpretations.
• Mechanisms such as distributed representations, trajectory amplification, and relational composition drive unsafe interpretations in multimodal systems.
• Defensive strategies like SRPO, MiShield, and generative interventions aim to align reasoning paths and mitigate SSUI’s compositional risks.

Searching arXiv for papers relevant to Safe-Semantics-but-Unsafe-Interpretation (SSUI) and closely related formulations in multimodal safety, text-to-image safety, and formal semantics. Safe-Semantics-but-Unsafe-Interpretation (SSUI) denotes a safety failure in which inputs that are individually benign, locally benign, or superficially benign are jointly interpreted in a way that yields harmful behavior. The explicit label is introduced for multimodal LLMs (MLLMs) in "When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal LLMs" (Cai et al., 15 Sep 2025), where safe image-text pairs can produce unsafe outputs through implicit reasoning risk. Closely related phenomena appear under other names across adjacent literatures: safe images combined with safe images and prompts can jailbreak LVLMs (Cui et al., 2024); multi-image implicit toxicity arises when each image is benign in isolation but harmful semantics emerge jointly (Lv et al., 1 Jul 2026); and text-to-image systems can interpret prompts unsafely when malicious semantics are distributed across an entire token sequence rather than localized in explicit unsafe words (Tan et al., 23 Mar 2026). Taken together, these works treat SSUI as a compositional, context-sensitive, and often trajectory-dependent failure mode rather than a simple failure of lexical filtering.

1. Definition and conceptual scope

In the narrow sense used by (Cai et al., 15 Sep 2025), SSUI is a cross-modal safety condition in which the image alone is safe, the text alone is safe, but the combined interpretation is unsafe. The paper formalizes the task as reasoning over a sequence of states

$\tau = (v_0 \to v_1 \to \dots \to v_T),$

with a verification function

$$\mathcal{F}(\tau) = \begin{cases} 1, & \text{if } A \in v_T\ 0, & \text{otherwise}. \end{cases}$$

The central claim is that safety alignment must hold not only at the final answer but throughout the reasoning trajectory (Cai et al., 15 Sep 2025).

Related work generalizes the same pattern beyond that formal definition. In LVLM jailbreaks, "a safe image can be exploited to achieve the same jailbreak consequence when combined with additional safe images and prompts," and the attack is attributed to "universal reasoning capabilities" and a "safety snowball effect" (Cui et al., 2024). In image-only moderation, multi-image implicit toxicity is defined as toxicity from the combination of benign images, with unsafe meaning emerging only when images are interpreted jointly (Lv et al., 1 Jul 2026). In text-to-image safety, DTVI does not introduce the SSUI term, but it explicitly targets cases where unsafe meaning is distributed across the full prompt sequence and therefore escapes token-level defenses (Tan et al., 23 Mar 2026).

This broader usage suggests that SSUI is not tied to one modality or one architecture. A plausible synthesis is that SSUI names a class of failures in which local semantic safety does not survive global model interpretation. The "safe" side of the contrast may occur at the level of tokens, images, unimodal inputs, intermediate views, interfaces, or language labels; the "unsafe interpretation" side appears when a downstream encoder, reasoner, optimizer, or runtime composes those inputs into a harmful latent state or action.

2. Mechanisms of unsafe interpretation

A recurring mechanism is distributed representation. DTVI argues that malicious semantics in modern jailbreak or red-team prompts are often "not confined to a few isolated tokens, but rather spread across the entire token sequence," because Transformer self-attention contextualizes each token embedding with the whole prompt (Tan et al., 23 Mar 2026). SafeRoPE makes an analogous claim for rectified-flow transformers: unsafe semantics concentrate in "interpretable, low-dimensional subspaces at head level," and unsafe generation depends on a sparse set of safety-critical heads that extract and propagate those directions, especially under multi-token interactions (Yang et al., 2 Apr 2026). In both cases, no single local feature need be obviously unsafe.

A second mechanism is trajectory-level amplification. SSA shows that an initial response may be only weakly unsafe or ambiguously aligned with harmful intent, yet subsequent prompts exploit that accepted context to elicit progressively more harmful outputs (Cui et al., 2024). UVR describes an analogous process inside multimodal diffusion transformers: a task-independent start-up stage first localizes unsafe output patches, followed by semantic amplification and interference stages in which harmful signals are propagated and entangled with benign content through multimodal attention (Yang et al., 5 Jun 2026). 3D editing systems extend this further: unsafe cues in edited 2D views can be absorbed into a 3D Gaussian representation and then rendered as coherent unsafe content across viewpoints (Meng et al., 14 May 2026).

A third mechanism is relational composition. MIIT identifies three key challenges for detection: individually benign inputs, distributed cues, and risky entity grounding. The risk is not simply a bag of suspicious features; it arises from "specific visual entities and their correlated relations" across images (Lv et al., 1 Jul 2026). The SSUI dataset for MLLMs makes the same point in reasoning form: harmfulness may emerge only after several inferential steps over a safe image-text pair (Cai et al., 15 Sep 2025).

There are also formal analogues outside AI safety. Secure-compilation work for unsafe languages models a component as protected until it encounters undefined behavior, after which it is treated as attacker-controlled but interface-bounded (Abate et al., 2018). Work on memory-safe languages and Rust unsafe encapsulation similarly shows that a language or API can look safe while operational safety depends on hidden unsafe code, FFI, constructors, or undocumented safety obligations (Barney et al., 2024, Rao et al., 2024, Rao et al., 30 Apr 2025). These are not multimodal SSUI instances, but they formalize the same structural contrast between safe-facing semantics and unsafe underlying interpretation.

3. Multimodal reasoning failures

The most direct multimodal SSUI demonstrations come from LVLM and MLLM safety. SSA operates in a black-box setting where the attacker can supply only safe images and text prompts across multiple turns. Its core result is that nearly any image can become a jailbreak seed once the agent retrieves or generates auxiliary benign images and the model reasons over the resulting multimodal context (Cui et al., 2024). On GPT-4o, direct harmfulness on SafeAttack-Bench averaged 0.67, initial-response-only reached 2.45, and full SSA reached 4.45; the paper further reports that harmfulness from safe images, 4.45, was nearly identical to harmfulness from unsafe images, 4.48 (Cui et al., 2024). This indicates that unsafe output can be a property of global composition rather than atomic input content.

MIIT studies the same problem in an image-only setting. Its dataset contains 1,434 unsafe multi-image instances, paired with an equal number of verified safe instances, across seven categories: Gore and Disturbing Content, Regulated Goods, Sexual Content, Violence and Conflict, Financial and Economic Crime, Self-Harm, and Hate and Extremism (Lv et al., 1 Jul 2026). The model MiShield-8B, trained with progressively distilled reasoning supervision, reaches 91.11 overall accuracy, with Unsafe $F_1 = 91.19$ and Safe $F_1 = 91.04$, outperforming representative commercial moderation APIs and larger general-purpose models (Lv et al., 1 Jul 2026). The paper’s ablations show that entity grounding matters most, but relation reasoning also matters, which supports the view that SSUI is fundamentally a grounded compositional reasoning problem.

The SSUI dataset of (Cai et al., 15 Sep 2025) makes this reasoning structure explicit. It contains 4,779 samples organized under 3 primary categories, 19 secondary categories, and 68 tertiary categories. The flagship example is a safe image of cleaning agents plus a safe request to clean a bathroom, where the hidden hazard is chlorine gas production. SRPO then optimizes branch-level reasoning preferences rather than only final responses. On the reported benchmark aggregate, Qwen2.5-VL-7B improves from Average $\uparrow = 47.58$ and Average $\downarrow = 33.51$ to 76.18 and 12.67 after SRPO training (Cai et al., 15 Sep 2025). This suggests that SSUI is not only a data problem but also a reasoning-path alignment problem.

4. Generative-model manifestations

Text-to-image systems exhibit a prompt-side form of SSUI when unsafe meaning is contextual, implicit, or distributed. DTVI frames its textual intervention around the full prompt embedding sequence $\mathbf{X}$, not selected token vectors, and then adds denoising-stage suppression of unsafe cross-attention features $\mathbf{h}^{t,l}$ (Tan et al., 23 Mar 2026). Its motivating example, "a woman, renaissance style, art," is important because the literal phrase is largely benign while the model’s learned visual priors can still imply nude portraiture. The paper therefore treats SSUI-like cases as failures of representation-level interpretation rather than failures of keyword detection.

Transformer-based diffusion models exhibit a finer-grained variant. SafeRoPE reports that subject phrases alone, such as "nude girl," rarely cause unsafe outputs, but contextual templates and modifiers substantially increase jailbreak success rates (Yang et al., 2 Apr 2026). Unsafe generation therefore depends on how query-key geometry in specific attention heads reads out risky directions from contextualized embeddings. UVR reaches a similar conclusion from an information-flow perspective: unsafe output patches first emerge during a start-up stage and are later amplified by $O_{\text{img} \to O_{\text{img}}$ or persist under $R_{\text{img} \to O_{\text{img}}$ interference in editing tasks (Yang et al., 5 Jun 2026).

The 3D case shows that SSUI can be cross-stage rather than merely cross-modal. In text-driven 3D Gaussian Splatting editing, 2D diffusion safety guidance alone does not reliably prevent unsafe 3D edits, because residual unsafe semantics in edited views are propagated and optimized across views into a persistent 3D representation (Meng et al., 14 May 2026). This suggests that "safe intermediate renderings" and "safe final representations" are not equivalent safety notions.

Concept-erasure work exposes a related tension between safety and semantic fidelity. PAIR argues that null-space removal methods fail because they erase unsafe concepts without guiding the model toward semantically aligned safe alternatives, thereby damaging structure, layout, and identity (Kim et al., 5 Feb 2026). Its unsafe-safe pairing formulation treats safe output not as unconditional absence but as counterfactual semantic realignment. This is an SSUI-relevant distinction: preventing unsafe interpretation may require a matched safe interpretation rather than simple suppression.

5. Evaluation, benchmarks, and formal criteria

Several recent benchmarks make SSUI measurable rather than anecdotal.

Benchmark or formalism	Setting	Core measure
SSUI / RSBench (Cai et al., 15 Sep 2025)	Safe image-text pairs with latent unsafe implication	Safety Rate, Effectiveness Rate, Safety-and-Effectiveness Rate
SafeAttack-Bench (Cui et al., 2024)	Safe images used to jailbreak LVLMs	Harmfulness score from 0 to 5
MIIT-dataset (Lv et al., 1 Jul 2026)	Benign images jointly implying unsafe semantics	Precision, Recall, $$\mathcal{F}(\tau) = \begin{cases} 1, & \text{if } A \in v_T\ 0, & \text{otherwise}. \end{cases}$$0, Accuracy
SafeGround (D'Incà et al., 14 Mar 2025)	Safe/unsafe image-text tuples for VLM evaluation	Pairwise safe preference, modality consistency, safe-input consistency

RSBench explicitly scores reasoning-path safety and usefulness: $$\mathcal{F}(\tau) = \begin{cases} 1, & \text{if } A \in v_T\ 0, & \text{otherwise}. \end{cases}$$1 and

$$\mathcal{F}(\tau) = \begin{cases} 1, & \text{if } A \in v_T\ 0, & \text{otherwise}. \end{cases}$$2

This matters because SSUI failures are often failures of reasoning trajectories rather than only final outputs (Cai et al., 15 Sep 2025).

SafeAttack-Bench uses a harmfulness score from 0 to 5 and validates evaluator reliability with 96.6% agreement for jailbreak success rate and 92.1% agreement for harmfulness scores (Cui et al., 2024). MIIT-dataset uses balanced safe and unsafe sets so that models can be penalized both for under-composition and over-composition, and reports that OpenAI Omni$$\mathcal{F}(\tau) = \begin{cases} 1, & \text{if } A \in v_T\ 0, & \text{otherwise}. \end{cases}$$3 detects only 16.38 Unsafe recall in the main evaluation, with the introductory pilot figure given as 16% when images are concatenated (Lv et al., 1 Jul 2026).

For text-to-image safety, DTVI defines Defense Success Rate as

$$\mathcal{F}(\tau) = \begin{cases} 1, & \text{if } A \in v_T\ 0, & \text{otherwise}. \end{cases}$$4

where $$\mathcal{F}(\tau) = \begin{cases} 1, & \text{if } A \in v_T\ 0, & \text{otherwise}. \end{cases}$$5 is the number of unsafe images from the undefended model and $$\mathcal{F}(\tau) = \begin{cases} 1, & \text{if } A \in v_T\ 0, & \text{otherwise}. \end{cases}$$6 is the number after defense (Tan et al., 23 Mar 2026). For VLMs, SafeGround isolates safety preference from retrieval correctness through pairwise comparisons such as

$$\mathcal{F}(\tau) = \begin{cases} 1, & \text{if } A \in v_T\ 0, & \text{otherwise}. \end{cases}$$7

so that degradation on safe inputs becomes directly visible (D'Incà et al., 14 Mar 2025).

6. Defensive strategies

Current defenses address different SSUI mechanisms rather than the entire phenomenon. SRPO targets reasoning-path misalignment by optimizing positive and negative branch continuations at each step; MiShield targets grounded compositional detection through entity grounding, cross-image correlations, and holistic safety deduction (Cai et al., 15 Sep 2025, Lv et al., 1 Jul 2026). These methods are strongest where unsafe meaning is primarily inferential.

Generative defenses intervene deeper in the model. DTVI combines sequence-level textual intervention with unsafe visual feature suppression and reports an average DSR of 94.43% across sexual-category benchmarks and 88.56 across seven unsafe categories; its ablations show 54.19% for textual-only, 77.57% for visual-only, and 92.99% for the joint system (Tan et al., 23 Mar 2026). SafeRoPE rotates unsafe-aligned subspaces only in safety-critical heads and reports that on FLUX.1-dev for nude prompts, Unsafe-1K unsafe rate drops from 38.8 to 15.4 while CLIP stays at 31.30 versus 31.31 baseline and VQA improves from 87.5 to 88.7 (Yang et al., 2 Apr 2026). UVR restricts unsafe information flow in MM-DiT and reports 91% erase rate in synthesis and 77% in editing (Yang et al., 5 Jun 2026). 3DEditSafe regularizes the 3D representation directly, reducing average view-level ASR from 0.347 to 0.190 and scene-level ASR from 0.722 to 0.389 (Meng et al., 14 May 2026).

Representation-preserving approaches address the opposite risk: making a model safer on unsafe inputs while less safe on safe inputs. SafeGround shows that Safe-CLIP raises unsafe-input safety but degrades safe-input safety, with $$\mathcal{F}(\tau) = \begin{cases} 1, & \text{if } A \in v_T\ 0, & \text{otherwise}. \end{cases}$$8 falling from 73.1 to 50.1 and $$\mathcal{F}(\tau) = \begin{cases} 1, & \text{if } A \in v_T\ 0, & \text{otherwise}. \end{cases}$$9 from 67.5 to 45.9, while UWM largely preserves safe-input behavior, reaching $F_1 = 91.19$0 and $F_1 = 91.19$1 (D'Incà et al., 14 Mar 2025). PAIR addresses a similar problem for diffusion concept erasure by mapping unsafe concepts to semantically aligned safe anchors rather than pushing them toward the unconditional null target; on nudity removal it reports ASR-IG 2.67, FID 16.93, Consistency 75.11, and HM 66.83 (Kim et al., 5 Feb 2026). These results suggest that SSUI defenses are often most effective when they preserve benign semantics while selectively blocking unsafe interpretation pathways.

7. Limitations and unresolved questions

No current method covers the full SSUI space. DTVI explicitly notes weaker coverage for deep pragmatics, cultural euphemism, multilingual coding, metaphor, and role-play frames, because its unsafe directions are learned from predefined harmful categories and unsafe-safe pairs (Tan et al., 23 Mar 2026). MIIT acknowledges that generated or controlled data may not match real social-media distributions, that cultural specificity is under-covered, and that boundary cases remain difficult (Lv et al., 1 Jul 2026). 3DEditSafe is evaluated on only 30 prompt-scene pairs and relies mainly on CLIP-based unsafe alignment, while UVR is strongest for relatively context-insensitive concepts and is less obviously sufficient for harms that are global, relational, or socially contextual (Meng et al., 14 May 2026, Yang et al., 5 Jun 2026).

There is also a persistent calibration problem. MiShield reports both over-composition false positives and benign reinterpretation false negatives (Lv et al., 1 Jul 2026). SafeGround shows that a model can improve on unsafe queries while becoming less safe on safe ones, so evaluating only attack success or refusal is insufficient (D'Incà et al., 14 Mar 2025). SSA shows that models can later recognize their own unsafe outputs when given extra reasoning steps, yet fail to block them during ordinary generation, which suggests a mismatch between generation-time safety and reflection-time safety (Cui et al., 2024).

A broader controversy concerns whether nominally safe abstractions are enough. Work on unsafe impedance argues that migration to memory-safe languages is necessary but insufficient because unsafe code can still be written, loaded, wrapped, or hidden behind safe interfaces, and proposes Unsafe Acceptance Processes to raise the practical barrier to unsafe composition (Barney et al., 2024). Secure-compilation work similarly formalizes safe semantics as conditional on the absence of undefined behavior rather than absolute (Abate et al., 2018). These analogues suggest that SSUI is not merely a quirk of multimodal foundation models. It is a general pattern in which local safety claims, interface labels, or aligned components remain vulnerable when downstream interpretation, composition, or compromise is not itself safety-aligned.