Stable Diffusion v1.5 Overview

• Stable Diffusion v1.5 is a latent diffusion model that converts images into latent space using a VAE and generates outputs via a U-Net denoiser with classifier-free conditioning.
• It incorporates Degeneration-Tuning, a method that scrambles specific prompts to block unwanted content while preserving overall image fidelity.
• Quantitative metrics like FID and IS validate DT’s capability to erase targeted concepts effectively without significant degradation on standard benchmarks.

Stable Diffusion v1.5 (SD-1.5) is a widely used latent diffusion model (LDM) for text-to-image synthesis. It employs a pre-trained variational autoencoder (VAE) to encode RGB images into a latent space, and a U-Net denoiser with cross-attention, facilitating conditional image generation via natural language prompts. SD-1.5 is distinguished by its “classifier-free” conditioning mechanism, leveraging fixed text encoders to guide generation. Owing to its large and diverse training set, SD-1.5 can synthesize images of arbitrary concepts, including those corresponding to intellectual property (IP), human faces, and distinctive artistic styles—a property that brings considerable utility, but also raises legal and ethical challenges. Recent works have sought to control or erase specific concepts within SD-1.5, with Degeneration-Tuning (DT) providing a parameter-level finetuning strategy that disables selected prompts while minimally degrading general generation quality (Ni et al., 2023).

1. Architectural Overview of Stable Diffusion v1.5

SD-1.5 builds on a latent diffusion paradigm, reducing the computational cost relative to pixel-space generative diffusion models. Its architecture comprises:

• Pre-trained VAE (ε): Encodes an image $x_0 \in \mathbb{R}^{H\times W \times 3}$ into a latent $z_0 = \varepsilon(x_0)$ and decodes $z\mapsto x$.
• U-Net denoiser ($\epsilon_\theta$): Predicts noise at each timestep $t$ on latent $z_t$. Receives conditioning from text-embedding $c$ via cross-attention layers.
• Text embedding ($\tau_\theta$): A frozen text encoder processes prompt tokens, injecting semantic guidance into the U-Net through cross-attention.
• Sampling Process: Initial latent $z_T \sim \mathcal{N}(0,I)$ is iteratively denoised through $T$ steps, employing the U-Net, ultimately yielding $z_0$, which is decoded to the final image.

The core loss for denoising during SD-1.5 training is:

$$L_{LDM} = \mathbb{E}_{z_0,c,t,\epsilon_t\sim\mathcal N(0,I)}\big[\|\epsilon_t - \epsilon_\theta(z_t, t, \tau_\theta(c))\|_2^2\big],$$

with $z_t = \alpha_t z_0 + \sigma_t \epsilon_t$ for $t = 1, \dotsc, T$.

2. Content Control and the Degeneration-Tuning Framework

The capacity of SD-1.5 to synthesize images associated with unwanted, sensitive, or IP-protected concepts necessitates robust methods for content restriction. Negative Prompting is commonly employed but exhibits fundamental limitations: if the prompt closely matches a banned concept, negative prompts often fail to erase the associated image features. Degeneration-Tuning (DT) addresses these deficiencies by training the U-Net to map selected prompts to "meaningless content"—via patchwise scrambling—while maintaining overall generative fidelity (Ni et al., 2023).

DT’s methodology involves:

• Scrambled-Grid images ($x_{sg}$): Created by segmenting an SD-generated image for concept $c_{sp}$ into an $S\times S$ patch grid (optimal $S=16$), then randomly permuting patches to destroy low-frequency structure.
• Anchor images ($x_{ac}$): Generated using an empty prompt $c_N$ to reinforce model stability on non-target concepts.
• Combined training set: $X = \{x_{sg}\} \cup \{x_{ac}\}$ with corresponding prompts $\{c_{sp}\} \cup \{c_N\}$.
• Tuning objective:

$$L_{DT} = \mathbb{E}_{x\in X,c,t,\epsilon_t\sim\mathcal N(0,I)}\big[\|\epsilon_t - \epsilon_\theta(x_t, t, \tau_\theta(c))\|_2^2\big].$$

The U-Net weights $\theta$ are updated such that input $c_{sp}$ leads to "degenerate" outputs, mapped to the easy-to-fit, scrambled-image space.

3. Training Protocol, Hyperparameters, and Model "Grafting"

DT operates as a minimal-overhead finetuning protocol. For each blocked concept:

• Dataset construction: 800–1000 scrambled images and ∼1200 anchor images.
• Hardware: Single node with 8×V100 GPUs.
• Batch size: 16.
• Learning rate: $1 \times 10^{-7}$ (contrasted with $1 \times 10^{-4}$ for full SD finetuning).
• Epochs: 60.
• Updated parameters: Only the complete U-Net is modified.

After DT, the updated U-Net can replace (“graft onto”) the U-Net in other conditional diffusion extensions such as ControlNet, yielding "Con-DT" models. These variants inherit the concept shielding property across further modalities, including pose and edge control.

4. Scrambled Grid Mechanism and Ablation Insights

The Scrambled Grid operator, $O(x)$, fragments the image into $S\times S$ patches and permutes them via a random mapping $\pi$, i.e., $O(x)[i,j] = P_{\pi(i,j)}(x)$. This operation disrupts global structure while minimally affecting local high-frequency detail, fostering rapid convergence during DT. Critical ablation findings include:

• Without Scrambled-Grid: Finetuning directly on real images or pure-color noise either fails to block the concept or collapses the model’s text understanding.
• Partial-U-Net Updates: Restricting DT to only cross-attention or ResBlock layers severely damages fidelity.
• Grid size: $16\times16$ emerges as the optimal balance between destructive capacity and learnability; $8\times8$ leaves residual structure, while $32\times32$ is excessively destructive.
• Anchor-to-Scrambled Ratio: Approximately 1:1 provides the best overall performance and generalization.

5. Quantitative and Qualitative Outcomes

DT delivers pronounced increases in FID and reductions in IS on targeted content with minimal collateral impact. Notably:

DT target	FID (targeted)	IS (targeted)	FID (COCO-30K)	IS (COCO-30K)
Original SD-1.5	—	—	12.61	39.20
“spider-man”	385.38	1.77	12.64	38.77
“Monet”	355.20	1.81	12.60	39.12
Joint DT	391.54	1.73	13.04	38.25

DT outperforms prior erasure methods (e.g., SLD, Erase) in preserving overall FID/IS on COCO-30K while achieving parameter-level concept removal.

Qualitatively, DT ensures:

• No recognizable content for the targeted prompt, even across multi-object or compositional contexts.
• Blocking of artistic styles without affecting generic photographic rendering.
• Independence of neighboring tokens; e.g., DT on "spider-man" does not impair generation for "spider" or "man".
• Consistent shielding when DT is transferred via grafting to control nets (pose, edge modalities).

6. Limitations, Drift, and Future Directions

DT exhibits robust prompt-specific blocking with minimal general degradation for single- or few-concept deployments. However, significant challenges persist:

• Continual application: Sequential DT on multiple concepts causes “butterfly effect” drift (e.g., FID on COCO-30K increases from 13.04 to 15.32 after multiple iterations; IS drops from 38.25 to 35.71), evidencing accumulating model degradation.
• Catastrophic forgetting: Anchor samples mitigate but do not eliminate drift; the open problem remains to balance continued concept erasure with stability.
• Highly structured or fine-grained targets: Some concepts or identities resist degenerate mapping, necessitating more advanced transformations or greater regularization.
• Scaling and automation: Real-time adaptation to new concept blocks, especially as legal and social norms evolve, remains an active area.
• Alternative degradation transforms: Superpixel shuffling, spectral filtering, or learned destruction networks offer potential avenues.
• Formal guarantees: Relating DT to cryptographic, privacy, or watermarking frameworks may yield provable assurances.

A plausible implication is that advancing continual, on-the-fly DT without loss of generalization will require both algorithmic innovation and theoretically grounded approaches to stability and selective forgetting.

7. Comparative Assessment and Significance

DT represents a lightweight, effective strategy for parameter-level concept erasure in SD-1.5, exceeding prior approaches both quantitatively (as measured by FID/IS/CLIP scores) and qualitatively. Its ability to operate at the model-weight level, transfer across conditional frameworks, and selectively affect only the intended concepts, positions DT as an impactful method for responsible deployment and control of large diffusion models. The underlying mechanism—associating unwanted prompts with “scrambled” outputs—demonstrates a general paradigm for redirecting model capacity in high-dimensional generative networks. Future exploration will likely pursue formal analyses, scalable deployments, and integration with evolving copyright and responsible AI frameworks (Ni et al., 2023).