SAFREE: Adaptive Safeguard for Diffusion Models

• SAFREE is a training-free and adaptive safeguard protocol that mitigates unsafe content in text-to-image and text-to-video generation without model retraining.
• It uses subspace analysis and dynamic filtering of prompt embeddings to identify and steer toxic content away while maintaining semantic fidelity.
• The system integrates Fourier-domain latent re-attention and self-validating filtering to achieve significant safety improvements across multiple generative backbones.

SAFREE is a training-free and adaptive safeguard protocol for safe text-to-image (T2I) and text-to-video (T2V) generation in modern generative diffusion models. Its distinguishing feature is that it does not require any model retraining or weight modification; instead, it operates by subspace analysis and dynamic filtering of prompt embeddings and latent features. SAFREE is designed to attenuate the risk of generating unsafe content—such as nudity, violence, or copyrighted styles—while preserving the intended semantic fidelity and visual quality of the outputs. The system applies to a wide family of generative backbones and can generalize to various safety categories and media modalities (Yoon et al., 2024).

1. SAFREE Architecture and Operational Pipeline

SAFREE operates as a modular guard that interfaces with inference-time pipelines of diffusion models. The protocol conducts the following key steps: (1) identification of a toxic concept subspace within the text embedding space constructed from user-defined keywords (e.g., "nudity", "violence"); (2) detection of prompt tokens whose excision most increases orthogonality to the toxic subspace; (3) projection of selected tokens away from the toxic subspace within the span of the original prompt embeddings; (4) application of a self-validating filtering mechanism that adaptively schedules detoxified embeddings during denoising, and (5) adaptive re-attention in latent visual space via Fourier-domain attenuation of low-frequency features linked to toxic concepts. The complete output is synthesized by the unmodified pretrained diffusion model conditioned on the filtered embeddings or latent features, ensuring safety without degradation of unrelated content.

2. Toxic Concept Subspace Detection and Token Identification

Given the D-dimensional CLIP or equivalent embedding space, the toxic subspace is spanned by the set $\mathcal{C}\in \mathbb{R}^{D\times K}$ of user-defined toxic keywords. For each token $i$ in a prompt $p=[e_0,\dots,e_{N-1}]$, SAFREE computes a pooled masked vector $\bar p_{\backslash i}$ (the mean embedding of all tokens except $i$):

$$\bar p_{\backslash i} = \frac{1}{N-1} \sum_{j\neq i} e_j$$

The projection operator $$P_{\mathcal{C}} = \mathcal{C}(\mathcal{C}^\top \mathcal{C})^{-1}\mathcal{C}^\top$$ is constructed. For each $\bar p_{\backslash i}$, the residual $$d_{\backslash i} = (I - P_{\mathcal{C}})\bar p_{\backslash i}$$ quantifies orthogonality to the toxic subspace. The norm $D_i = \|d_{\backslash i}\|_2$ is used to flag tokens whose removal best distances the prompt from toxicity. Token $i$ is masked ($m_i=1$) if $D_i$ exceeds $(1+\alpha)$ times the mean of $D_j$ over $j\neq i$, with $\alpha=0.01$ (Yoon et al., 2024).

3. Subspace Orthogonalization and Embedding Steering

SAFREE projects the prompt embedding $p$ away from the toxic concept subspace while maintaining coherence within the span of the original prompt. The input subspace matrix $\mathcal{I}$ comprises all pooled masked embeddings, and its projector $P_{\mathcal{I}}$ is constructed:

$$\mathcal{I} = [\,\bar p_{\backslash0},\dots,\bar p_{\backslash(N-1)}]$$

$$P_{\mathcal{I}} = \mathcal{I}(\mathcal{I}^\top \mathcal{I})^{-1}\mathcal{I}^\top$$

The full prompt is jointly steered as:

$$p_{\text{proj}} = P_{\mathcal{I}} (I - P_{\mathcal{C}}) p$$

The filtered embedding $p_{\text{safe}}$ is computed element-wise:

$$p_{\text{safe}} = m \odot p_{\text{proj}} + (1-m) \odot p$$

where $m$ is the binary mask for detoxification and $\odot$ denotes broadcast multiplication.

4. Self-Validating Filtering and Adaptive Injection During Denoising

Different denoising timesteps in a diffusion process have variable influence on the synthesis of toxic content. SAFREE calculates a step threshold $t'$ via:

$$t' = \gamma\, \mathrm{sigmoid}\left(1-\cos(p,p_{\text{proj}})\right), \quad \gamma=10$$

At each denoising step $t$, the prompt conditioning is chosen as:

$$p_{\text{SAFREE}}(t) = \begin{cases} p_{\text{safe}}, & t \leq \mathrm{round}(t') \ p, & \text{otherwise} \end{cases}$$

When the prompt is close to the toxic subspace, early diffusion steps use the detoxified embedding, increasing safety intervention. For prompts already distant from toxicity, most steps use the original embedding, preserving quality.

5. Adaptive Latent Re-attention via Fourier Filtering

In the latent visual space of the denoising UNet, SAFREE applies a selective attenuation mechanism leveraging the Fourier transform. For latent feature maps $h(p)$ (original) and $h(p_{\text{SAFREE}})$ (filtered), SAFREE computes masked Fourier maps:

$$\mathcal{F}(p) = b \odot \mathrm{FFT}(h(p)),\quad \mathcal{F}(p_{\text{SAFREE}}) = b \odot \mathrm{FFT}(h(p_{\text{SAFREE}}))$$

where $b$ selects the central low-frequency components. The re-weighted map $\mathcal{F}'$ is constructed by replacing entries where the filtered component is larger than the original with a scaled version:

$$\mathcal{F}_i' = \begin{cases} s\,\mathcal{F}(p_{\text{SAFREE}})_i, & \mathcal{F}(p_{\text{SAFREE}})_i\ >\ \mathcal{F}(p)_i \ \mathcal{F}(p_{\text{SAFREE}})_i, & \text{otherwise} \end{cases}$$

Parameters $s_1=0.9$, $s_2=0.2$ are used for early and late UNet blocks, respectively. The inverse FFT reconstructs the re-weighted latent features, which are then injected at each block during denoising.

6. Empirical Performance and Extensibility

SAFREE demonstrates strong performance across multiple benchmarks and toxic concept categories, including I2P, MMA-Diffusion, artist-style removal, and SafeSora video datasets. Safety is quantified via attack success rate (ASR), while quality is assessed with FID, CLIP, TIFA, and LPIPS. Typical results include reductions in ASR of 22% absolute (e.g., ASR≈0.034 vs. 0.115 for SLD-Max on nudity attacks) with FID scores maintained at 36.35, CLIP at 31.1, and TIFA at 0.790. For video models (ZeroScopeT2V, CogVideoX‐5B), SAFREE reduces unsafe concept rates by 20–40 points for various categories. The protocol extends zero-shot to SDXL, SD-v3, and T2V backbones with consistent gains. Inference time is approximately 9.8 s/sample on an A6000 GPU (Yoon et al., 2024).

Ablation indicates that the mask threshold $\alpha$ and step scheduling parameter $\gamma$ are robust, and the Fourier domain re-attention successfully suppresses residual global-style toxicity. Artist-removal metrics, including LPIPS_e=0.42 and LPIPS_u=0.31, indicate selective erasure with minimal impact on untargeted content.

7. Implementation Details and Practical Considerations

SAFREE operates on standard CLIP text embeddings without architectural modifications to the base diffusion model. The only required inputs are the prompt tokens, the toxic keyword list, and the pretrained generative backbone. All filtering occurs at inference; no training or weight editing is performed. The latent re-attention utilizes a binary mask over Fourier coefficients, usually selecting the central 1/8 of the spectral map. 100 denoising steps are used in all reported experiments. For video, filtering is applied per-frame or by concatenated text embeddings as appropriate.

SAFREE is compatible with safety-critical applications where model weights must remain frozen (e.g., medical imaging, copyright-sensitive generation) and can be configured in a plug-and-play manner for both image and video synthesis workflows.

8. Context and Significance in Safe Generative AI

Prior unlearning, editing, and retraining paradigms for safe generation suffer from slow iteration, dependence on curated datasets, and risk of collateral degradation. SAFREE's methodology—subspace token filtering, adaptive injection, and frequency-domain latent modulation—addresses the need for instant response to emerging threats and user-specified censorship requests, with demonstrable generalization across backbones and modalities. This suggests that fine-grained safety interventions exploiting embedding geometry and spectral latent manipulation represent a scalable protocol for safeguarding open-domain generative models (Yoon et al., 2024). A plausible implication is that future safety regulation for generative AI may favor inference-layer interventions over model editing, reserving training-based schemes for global performance improvements rather than context-sensitive filtering.