AEGIS: A Mechanism-Guided Defense against Visual Synonym Jailbreaks in Text-to-Image Models

Abstract: Text-to-image diffusion models have achieved high visual fidelity and broad adoption, but remain vulnerable to safety violations when adversaries exploit them to synthesize illicit content. Existing alignment paradigms, from input sanitization to structural feature pruning, are largely organized around unsafe concepts explicitly exposed during filtering, editing, or localization. This leaves a blind spot for visual synonym attacks (VSA), a jailbreak where benign-looking prompts elicit prohibited imagery through implicit visual associations. As a result, current defenses face a safety-utility dilemma: they may either under-mitigate VSA threats or over-suppress visually similar benign concepts. The core challenge is that VSA hides the unsafe target at the textual surface while revealing it through generation-time visual-semantic convergence. In this work, we therefore shift from static suppression of pre-specified unsafe concepts to dynamic tracing of how unsafe semantics emerge during generation. Our mechanistic analysis shows that VSA and explicit unsafe prompts converge through sparse semantic-injecting attention heads, which serve as inference-time bottlenecks for prohibited visual semantics. Based on this insight, we propose AEGIS (Adaptive Evasion Guard via Identification and Steering), an inference-time defense that applies similarity-aware repulsion only at the identified vulnerable heads. Evaluated against 16 baselines, AEGIS improves both safety and utility. On SD 1.4, it reduces ASR to $\mathbf{0.00}/\mathbf{0.03}$ for in-domain violence/nudity VSA and achieves ASRs $\le \mathbf{0.09}$ on out-of-domain explicit and adversarial attacks. It preserves benign fidelity, avoids suppressing hard-negative concepts, and transfers to SD 2.1 and FLUX.1 after re-identifying the critical heads for each backbone.

• The paper proposes AEGIS, which leverages Lasso regression to identify ~10% of attention heads responsible for 70% of unsafe activations in text-to-image models.
• It employs anchor-based repulsion and similarity-aware adaptive gating to dynamically steer semantic pathways, mitigating unsafe generation while maintaining benign utility.
• Empirical evaluations across multiple backbones demonstrate near-zero attack success rates and preserved image fidelity, outperforming 16 state-of-the-art safety methods.

Mechanism-Guided Defense against Visual Synonym Jailbreaks in Text-to-Image Models

Introduction and Problem Formulation

Text-to-image (T2I) diffusion models have become pervasive due to their high-fidelity generation, but their susceptibility to adversarial misuse remains a critical challenge. Traditional safety interventions center on explicit and semantically proximate unsafe prompts, leveraging input sanitization, trigger-specific pathway disruption, or structural feature pruning. However, these approaches create a structural vulnerability: Visual Synonym Attacks (VSA)—wherein benign-seeming prompts (e.g., "red painting") are mapped to generation trajectories indistinguishable from genuinely prohibited prompts (e.g., "blood splatter") in image space. This semantic misalignment enables adversaries to circumvent both text-centric filters and editing strategies, while overzealous suppression often degrades benign, visually adjacent concepts, engendering a profound safety-utility trade-off.

Figure 1: (a) Unprotected generation of explicit, visual synonym, and benign prompts. Generation of visual synonyms resembles explicit prompts, despite their semantic orthogonality in text space. (b) Under-mitigation: Text-centric sanitizers and disruptions intercept explicit triggers but are bypassed by visual synonyms. (c) Over-mitigation: Structural pruning blocks attacks but severely degrades visually similar benign concepts. (d) Ours: Adaptively steers localized semantic pathways to defuse all attacks while preserving benign utility.

Mechanistic Analysis of Visual Synonym Attacks

The paper undertakes a systematic mechanistic investigation to localize the model components underlying this vulnerability. The generation process is profiled using anchor-based similarity signatures, tracing how inputs with starkly divergent semantics in textual embedding space can converge in the visual latent trajectory over the course of denoising. Analyses reveal that explicit triggers induce strong, early activations across the architecture, while visual synonyms activate a sparse set of deep, semantic-injecting attention heads at later diffusion steps.

Figure 2: The mechanism of VSA. (a) Conceptually, VSA transitions from a safe text space into an unsafe visual space during denoising. (b) This evasion exploits stealthy text-visual semantic misalignment, but is auditable via a malicious anchor. (c) Relative latent trajectories on three prompt sets show synonyms masquerading as benign early on, then dynamically converging to the explicit distribution in late steps.

To isolate the specific attention heads responsible, the authors employ Lasso regression-based attribution, confirming that less than 10% of attention heads account for the vast majority (≈70%) of unsafe semantic pathway activations.

Figure 3: Distribution of attention head importance (bloody violence, SD 1.4). Most heads have negligible importance, while a small subset (~10%) accounts for nearly 70% of the total unsafe semantic attribution, forming a concentrated pathway for the malicious payload.

Visualizations of attention in these critical heads further confirm spatial and semantic specificity: their highest activations consistently focus on regions and patterns correlated with unsafe image features, not observed for random heads.

Figure 4: Attention visualization of top and random heads.

Adaptive Evasion Guard via Identification and Steering (AEGIS)

Building on the mechanistic findings, the proposed defense—AEGIS (Adaptive Evasion Guard via Identification and Steering)—operates as a plug-and-play inference-time steering mechanism with three major components:

1. Precise Attribution: Only the sparsely identified critical attention heads (per-concept and per-architecture) are targeted for intervention, minimizing the perturbation footprint.
2. Anchor-Based Repulsion: Using an anchor embedding corresponding to an explicit forbidden concept, AEGIS applies a controlled repulsive update to the head activation vector when its similarity to the anchor crosses a threshold.
3. Similarity-Aware Adaptive Gating: The intervention's magnitude is proportional to both the activation strength and cosine similarity with the anchor, saturating only in high-risk (unsafe) scenarios.

This strategy ensures robust suppression of unsafe content even for stealthy VSAs while preserving utility (fidelity and semantic alignment) for benign and semantically ambiguous prompts.

Empirical Evaluation

AEGIS is evaluated on Stable Diffusion v1.4, v2.1, and FLUX.1-dev backbones across explicit, adversarial, and VSA benchmarks spanning nudity and violence. It is compared to 16 state-of-the-art safety alignments across all prevailing paradigms (semantic sanitization, pathway disruption, and structural pruning). Key results include:

• Attack Success Rate (ASR): On SD 1.4, AEGIS attains ASR of 0.00/0.03 for violence/nudity on VSA data, outperforming all baselines, with ≤0.09 ASR on out-of-domain explicit and adversarial jailbreaks.
• Utility Metrics: FID and CLIP scores remain comparable to the original model, and AEGIS avoids over-mitigation for hard-negative benign prompts similar in appearance to unsafe concepts.

  Figure 5: Qualitative comparison of baselines and our method. Our method effectively eliminates the nudity or violent content while preserving subject identity and scene layout.

  

  Figure 6: Qualitative comparison on benign visual synonyms. Safe prompts visually resembling: (top) bloody violence, e.g., chopping tomatoes; (bottom) nudity, e.g., mannequins.

Ablation and robustness analysis confirm that attribution precision and adaptive steering are key to balancing safety and utility—randomized interventions collapse utility, and simply retraining baselines on VSA data cannot match the trade-off achieved by AEGIS. Furthermore, adaptive (white-box) attacks that attempt to evade the identified steering heads fail to restore unsafe generations, indicating resilience at the mechanistic level.

Figure 7: Hyperparameter ablation study. We analyze the impact of injection ratio ρ, strength factor α, and adaptive gating threshold β on the safety-utility trade-off. The star denotes our selected configuration, which achieves an optimal balance (bottom-left).

Figure 8: Results of adaptive attacks under different δ.

Cross-architecture experiments show the head-attribution procedure and repulsion mechanism transfers directly to new backbones—after retracing critical heads—without material degradation in mitigation performance or utility.

Theoretical and Practical Implications

This work formally characterizes VSAs as a structural vulnerability rooted in the dynamic convergence of visual semantics, rather than static associative mappings in text-embedding space. The identification of sparse, semantically privileged attention heads as principal conduits for illicit concept injection reframes safety for generative models as a mechanism-level rather than representation-level problem. AEGIS demonstrates that targeted, inference-time modulation of these pathways can simultaneously achieve robust, precise safety alignment and avoid broad semantic collateral damage—breaking the historical safety-utility dilemma of T2I alignment.

The framework is computationally efficient, requiring only lightweight attribution model fitting and minimal per-generation overhead. Since the defense localizes and mitigates the actual causal pathway for semantic convergence, it is resilient to prompt engineering and adaptive attacks that attempt to obfuscate unsafe intent at the input or embedding level. The demonstrated transferability across architectures further supports that semantic injection is a model-agnostic structural phenomenon.

Future Directions

Mechanism-guided alignment can potentially extend to real-time, multi-concept regulation, automated discovery of emergent unsafe concepts, and compositional safety. The head-level steering paradigm may be generalizable to broader classes of multimodal and autoregressive generative models, as well as to domains beyond violence and nudity. Further research is warranted into scaling these techniques to open-domain settings, quantifying their effectiveness against novel attack vectors, and integrating interpretability with system-level deployment guarantees.

Figure 9: Qualitative comparisons on hard negative prompts. We evaluate models on safe prompts that are visually similar to unsafe concepts. (Top) Violence: Benign prompts involving red liquids. (Bottom) Nudity: Benign prompts involving skin-tone figures. Baselines often exhibit over-mitigation, damaging the semantics of safe images. Our method effectively preserves the visual details of benign concepts.

Conclusion

The AEGIS framework exemplifies a rigorous mechanism-guided realignment of T2I safety, moving the field beyond static filter-based and pruning approaches toward dynamic, minimally invasive, and theoretically grounded semantic regulation. By explicitly tracing and steering the root-cause pathways of unsafe content emergence, AEGIS addresses both under- and over-mitigation, offering a robust and practical blueprint for future AI safety-critical generative deployments.

---

Reference:

"AEGIS: A Mechanism-Guided Defense against Visual Synonym Jailbreaks in Text-to-Image Models" (2607.06120)