DeContext as Defense: Safe Image Editing in Diffusion Transformers (2512.16625v1)

Abstract: In-context diffusion models allow users to modify images with remarkable ease and realism. However, the same power raises serious privacy concerns: personal images can be easily manipulated for identity impersonation, misinformation, or other malicious uses, all without the owner's consent. While prior work has explored input perturbations to protect against misuse in personalized text-to-image generation, the robustness of modern, large-scale in-context DiT-based models remains largely unexamined. In this paper, we propose DeContext, a new method to safeguard input images from unauthorized in-context editing. Our key insight is that contextual information from the source image propagates to the output primarily through multimodal attention layers. By injecting small, targeted perturbations that weaken these cross-attention pathways, DeContext breaks this flow, effectively decouples the link between input and output. This simple defense is both efficient and robust. We further show that early denoising steps and specific transformer blocks dominate context propagation, which allows us to concentrate perturbations where they matter most. Experiments on Flux Kontext and Step1X-Edit show that DeContext consistently blocks unwanted image edits while preserving visual quality. These results highlight the effectiveness of attention-based perturbations as a powerful defense against image manipulation.

• The paper presents a novel adversarial defense that perturbs cross-attention in early denoising timesteps to block context-driven identity transfer.
• It employs a localized perturbation strategy that achieves a >70% drop in face recognition accuracy while maintaining high image quality.
• Extensive experiments across benchmarks demonstrate robust generalization to diverse prompts and object types with minimal visual degradation.

DeContext as Defense: Suppressing Contextual Leakage in Diffusion Transformers for Safe Image Editing

Motivation and Problem Statement

Recent advances in in-context diffusion transformers (DiTs) have dramatically improved the realism and flexibility of image editing, notably enabling precise identity transfer and stylistic modifications using only a single context image and textual prompts. However, such in-context conditioning presents unprecedented privacy risks: dissemination of images online exposes them to unauthorized identity cloning, impersonation, and malicious edits, effectively bypassing legacy data safeguards. Existing diffsion defense approaches targeting UNet architectures or latent/encoder perturbations are ineffective against context propagation through large-scale multimodal attention in contemporary DiT frameworks, often degrading image quality or failing to block identity extraction.

Mechanism Analysis: Attention as the Propagation Channel

The central insight of this work is that context image influence in DiT-based in-context editing models is exclusively mediated via multimodal cross-attention, specifically between context-key and target-query tokens during generation. Standard adversarial attacks (PGD-based perturbations maximizing denoising loss) are insufficient, yielding only minor lighting or blur artifacts without breaking the context’s semantic transfer. Conversely, direct intervention that zeros out cross-attention from target to context tokens reliably severs the context (Figure 1).

Figure 1: Standard PGD attack yields only cosmetic changes (center), whereas explicit attention suppression (right) eliminates context-associated identity transfer.

Systematic analysis further reveals that context propagation is concentrated at high diffusion timesteps (early denoising phases) and within the initial transformer blocks, with gradient magnitudes on context tokens peaking in the noisy initialization regime (Figure 2).

Figure 2: Context image gradients dominate during early denoising, confirming concentration of context propagation in high-noise timesteps.

The DeContext Method: Attention-Suppression via Localized Perturbation

DeContext implements a targeted adversarial defense that directly minimizes cross-attention activations between context-key and target-query tokens. The adversarial objective maximizes $1 - r_{\text{ctx}}$, where $r_{\text{ctx}}$ quantifies the mean cross-attention from target queries to context keys (averaged over heads and blocks). The context image is iteratively perturbed—under a visually imperceptible norm budget—by ascending the gradient of this loss, sampled across diverse prompts, timesteps, and random noise seeds to ensure prompt-agnostic and robust protection.

Optimization is concentrated in early denoising timesteps and the initial/mid transformer blocks, targeting the locus of maximal context information flow. This block-wise, timestep-wise localized attack achieves context decoupling and identity removal without modifying the generative model or significantly degrading image realism (Figure 3).

Figure 3: DeContext pipeline: context image is perturbed to minimize its attention activation, iteratively across blocks and high-noise timesteps for robust context suppression.

Quantitative and Qualitative Results

Experiments on FLUX-Kontext and Step1X-Edit (state-of-the-art DiT-based in-context editors) using VGGFace2 and CelebA-HQ benchmarks demonstrate DeContext’s superiority over existing protection baselines, including Anti-DreamBooth, AdvDM, CAAT, FaceLock, and naive PGD strategies. DeContext achieves:

• >70% drop in face recognition accuracy (as evaluated by ArcFace ISM and RetinaFace FDFR metrics), consistently outperforming previous defenses.
• Image quality metrics (BRISQUE, FID) within 20% of unperturbed images, notably preserving photorealism while eliminating identifying cues.
• Robust generalization across multiple editing prompts and on unseen item/object images, maintaining context suppression and artifact-free synthetic outputs.
• Consistent user study preference for DeContext over alternative defenses, both for identity removal and overall editing quality (Figure 4).

  Figure 5: Prior attacks introduce significant artifacts and fail to fully remove context identity. DeContext preserves visual quality while reliably suppressing context-based identity transfer.

  

  Figure 6: Qualitative results on Step1X-Edit verify prompt-agnostic context detachment and visual fidelity across diverse edits.

  

  Figure 4: User study rankings indicate a strong preference for DeContext in both privacy protection and output realism.

Ablation and Failure Modes

Ablation studies confirm that attack efficacy scales with perturbation budget and is most effective when applied to early and middle blocks. Excessive perturbation increases artifact risk. DeContext’s protection performance diminishes for scene-altering prompts where text guidance inherently dominates over context image cues (Figure 7).

Figure 7: Failure case: for prompts demanding dramatic scene change, even strong context suppression has limited effect as textual control predominates.

Extensions: Items and Generalization

Tests on object/item images confirm DeContext’s prompt-agnostic generalization: semantic similarity metrics (DINO, CLIP-I, SSIM) drop substantially after defense, indicating successful context decoupling across domains (Figures 9, 10).

Figure 8: Defense on human portraits: DeContext removes identity cues across varied editing directions.

Figure 9: Defense on item images: context suppression generalizes beyond faces to objects, eliminating context-driven similarity.

Implications and Future Directions

DeContext introduces an efficient, inference-time protection strategy uniquely tailored to in-context DiT-based editors, outperforming prior UNet-oriented and latent/encoder-based defenses. It demonstrates that precise, localized perturbation targeting attention pathways is both necessary and sufficient for robust privacy preservation in multimodal transformer diffusion pipelines.

Practical implications include improved identity privacy guarantees for public-facing image datasets, controllable content protection for creators, and enhanced input-level safety in vision AI systems. The theoretical contribution lies in exposing the role of context-key/target-query cross-attention as the unique vulnerability for context leakage, inviting further interpretability and defense research in transformer-based generative models.

Outstanding challenges involve scaling DeContext for extremely large models, enhancing black-box transferability (especially when internal attention states are inaccessible), and designing selective attention reduction strategies to handle complex scene modifications where context/text interplay must be dynamically modulated.

Conclusion

DeContext provides a systematic, attention-aware adversarial defense that robustly suppresses context-driven identity and content leakage in DiT-based in-context image editing models. By localizing perturbations to multimodal cross-attention pathways, it achieves strong protection-effectiveness and high visual fidelity, establishing a new paradigm for input-level privacy preservation in modern generative transformers (2512.16625).