StoryDiffusion: A Stable Diffusion XL Variant

• StoryDiffusion is a plug-in variant of Stable Diffusion XL that uses CSA and SMP to maintain subject consistency across image sequences and videos.
• It operates in a zero-shot manner by replacing self-attention layers without fine-tuning, repurposing pretrained diffusion backbone weights.
• Benchmarking shows improved CLIP similarity and user preference, indicating enhanced character coherence and smoother motion transitions compared to prior methods.

StoryDiffusion is a plug-in variant of Stable Diffusion XL (SD XL) designed to produce subject-consistent image sequences and smooth transition videos from text stories, addressing the long-standing issue of maintaining visual consistency for complex subjects across a generated sequence. The framework introduces Consistent Self-Attention (CSA) for training-free cross-frame content binding within image batches and a Semantic Motion Predictor (SMP) for generating temporally coherent intermediate video frames. StoryDiffusion operates as a zero-shot augmentation requiring no finetuning of the underlying diffusion backbone and remains fully compatible with pretrained SD XL or SD 1.5 weights (Zhou et al., 2024).

1. Architectural Foundations and Baseline Limitations

Stable Diffusion XL (SD XL) operates as a latent diffusion model. Images are mapped to and from a compact latent space $z \in \mathbb{R}^{H' \times W' \times C}$ via a pretrained Variational Autoencoder (VAE). Generation is handled by a U-Net denoiser $\varepsilon_\theta(z_t, t, c)$, where $z_t$ denotes the noisy latent, $t$ is the timestep, and $c$ is a text embedding from CLIP. Each U-Net block comprises (i) spatial self-attention, (ii) cross-attention to CLIP text tokens, and (iii) feed-forward layers. Sampling procedures such as DDIM or DDPM iteratively denoise random Gaussian latents into final image latents, which the VAE decoder transforms into images (e.g., $512 \times 512$ pixels).

SD XL generates images per prompt independently. Consequently, when tasked to generate a narrative or comic—i.e., a sequence of images involving recurring subjects—SD XL lacks a mechanism for cross-frame information flow. As a result, character identity, attire, and item persistency are not maintained, leading to subject drift across multiple images (Zhou et al., 2024).

2. Consistent Self-Attention Mechanism

Consistent Self-Attention (CSA) replaces every self-attention layer in the SD XL U-Net to facilitate subject coherence across a batch of related images. Given a batch of spatial features $$\mathcal{I} = \{I_1, \ldots, I_B\},\, I_i \in \mathbb{R}^{N \times C}$$, standard self-attention computes:

$$Q_i = W_q I_i, \quad K_i = W_k I_i, \quad V_i = W_v I_i,$$

$$O_i = \mathrm{softmax}\left(\frac{Q_i K_i^\top}{\sqrt{d}}\right)V_i.$$

CSA modifies this by sampling a subset of tokens $S_i$ from all images in the batch except $i$ using a random sampling rate $r$:

$$S_i = \mathrm{RandSample}\left(\{I_j\}_{j \neq i};\, r\right) \in \mathbb{R}^{N_s \times C}.$$

The key/value pool for each image is augmented as

$P_i = [I_i; S_i],$

which shares the original SD XL projection weights:

$K_{P_i} = W_k P_i, \quad V_{P_i} = W_v P_i.$

The modified self-attention becomes

$$O_i = \mathrm{softmax}\left(\frac{Q_i K_{P_i}^\top}{\sqrt{d}}\right)V_{P_i}.$$

This structure enables each image within a batch to aggregate context from specific spatial tokens of other images, enforcing consistency in recurring visual attributes—e.g., facial features, clothing, or props—across the generated sequence. Implementation ablations show subject consistency degrades for sampling rates below $r = 0.3$ (default $r \approx 0.5$). CSA is implemented in a tile-wise manner to control memory overhead (Zhou et al., 2024).

3. Zero-Shot Plug-In and Hyperparameterization

CSA functions as a hot-plug replacement for the self-attention layers in the SD XL U-Net. No weight finetuning or retraining is required; the original pretrained SD XL Q/K/V weights are reused in CSA. Inference consists of:

• Loading SD XL backbone weights;
• Replacing self-attention with CSA in all U-Net blocks;
• Running the DDIM sampler (typically 50 steps) with classifier-free guidance ($g \approx 5.0$);
• Setting the spatial token sampling rate at $r=0.5$ and limiting memory via tile size ($W \approx 8$).

A direct implication is rapid adaptation to arbitrary stories or image batch sizes without retraining (Zhou et al., 2024).

4. Semantic Motion Predictor for Video Synthesis

To extend subject-consistent generation to videos, StoryDiffusion introduces the Semantic Motion Predictor (SMP), which interpolates and refines semantic trajectories between keyframes in CLIP-embedding space to produce motion-consistent intermediates. The SMP pipeline comprises:

(a) Semantic Encoding: A pretrained CLIP image encoder ($E$) computes semantic embeddings for source and target frames: $K_s = E(F_s),\ K_e = E(F_e)$.

(b) Trajectory Initialization: Linear interpolation in semantic space generates a crude intermediate embedding sequence:

$$\widetilde{K}_i = (1 - \frac{i}{L})K_s + \frac{i}{L}K_e,\,\quad 1 \leq i \leq L$$

for $L$ in-between frames.

(c) Trajectory Refinement: An $M$-layer transformer ($B$) further processes these embeddings:

$$(P_1, \ldots, P_L) = B(\widetilde{K}_1, \ldots, \widetilde{K}_L),$$

training against real ground-truth frames via:

$$\mathcal{L}_{\rm SMP} = \frac{1}{L} \sum_{i=1}^L \|G_i - \hat{G}_i(P_i)\|_2^2,$$

where $\hat{G}_i(P_i)$ represents the diffusion decoder’s frame from conditioning $P_i$.

(d) Diffusion Conditioning: For each intermediate frame, the refined motion embedding $P_i$ is concatenated to the text embedding ($T$) at the U-Net cross-attention layers:

$$V_i' = \mathrm{CrossAttn}(V_i, [T; P_i], [T; P_i]).$$

This construction ensures that subject appearance and layout remain coherent throughout potentially large pose, viewpoint, or scene transitions (Zhou et al., 2024).

5. End-to-End Inference Workflow

The complete image and video generation pipeline consists of two sequential stages:

Stage 1 (Comics Generation):

1. Tokenize a multi-sentence story into prompts $\{c_1, \ldots, c_K\}$.
2. Form a batch of size $K$, run the DDIM sampler (50 steps) on SD XL with CSA.
3. Decode the resulting latents into $K$ consistent images.

Stage 2 (Video Synthesis):

1. For each adjacent image pair $(F^k, F^{k+1})$, encode semantic pairs: $(K_s, K_e) = (E(F^k), E(F^{k+1}))$.
2. Interpolate $\widetilde{K}_i$ and refine embeddings via the transformer $B$ to yield $\{P_i\}_{i=1}^L$.
3. For each $i$, generate video frame $\hat{G}_i$ conditioned on $[T; P_i]$.
4. Concatenate all segments to construct a complete, temporally smooth video (Zhou et al., 2024).

6. Empirical Performance and Benchmarking

Extensive benchmarking demonstrates StoryDiffusion's improvements in both qualitative and quantitative terms relative to prior baselines.

Subject-consistency in images (compared on SD XL backbone with 50 DDIM steps, $g = 5.0$):

• Text–image CLIP similarity: IP-Adapter 0.613, PhotoMaker 0.654, StoryDiffusion 0.659
• Character–character CLIP similarity: IP-Adapter 0.880, PhotoMaker 0.892, StoryDiffusion 0.895
• User preference: StoryDiffusion 72.8%, IP-Adapter 10.4%, PhotoMaker 16.8%

Transition video quality (compared on SD 1.5 backbone with 50 DDIM steps, $g = 7.5$):

Metric	SEINE	SparseCtrl	StoryDiffusion
LPIPS-first (↓)	0.433	0.491	0.379
LPIPS-frames (↓)	0.222	0.177	0.164
CLIPSIM-first (↑)	0.926	0.903	0.961
CLIPSIM-frames(↑)	0.974	0.976	0.987
User preference	11.6%	6.4%	82%

Qualitative observations show enhanced preservation of attire, facial appearance, and props, with minimal artifacts and physically plausible motion—robust even to large-scale pose or viewpoint changes (Zhou et al., 2024).

7. Implementation Details and Reproducibility

• Code and pretrained weights are available at https://github.com/StoryDiffusion/StoryDiffusion.
• Experiment configuration: SD XL, 50 DDIM steps, guidance 5.0, CSA sampling rate $r=0.5$, and tile size $W \approx 8$.
• SMP training uses WebVid10M transitions, integrating AnimateDiff V2 temporal module and CLIP ViT-H/14 encoder, with an 8-layer transformer (hidden dim 1024, 12 heads), optimized using AdamW ($\text{lr}=10^{-4}$, 100k iterations on 8×A100 GPUs).
• For story sequences longer than batch limits, CSA is applied in a sliding window, and SMP is applied pairwise.
• All components exert zero-shot compatibility with standard SD XL/1.5 backbones; full reproductions of subject-consistent image sequences and transition videos require only minimal code changes (Zhou et al., 2024).