Style-Prompting Guidance (SPG)

• Style-Prompting Guidance (SPG) is a training-free, sampling-time method that augments text-to-image diffusion by adding a style-specific branch.
• It isolates style as a directional deviation in noise space via a style-conditioned unconditional prediction, preserving CFG’s semantic contrast.
• SPG combines KV replacement with AdaIN to ensure effective stylization while maintaining content fidelity and robust text alignment.

Style-Prompting Guidance (SPG) is a training-free, sampling-time method for style-specific image generation in text-to-image diffusion models. It addresses a central limitation of standard Classifier-Free Guidance (CFG): although CFG improves semantic fidelity to a text prompt, controlling visual style from a single reference image without sacrificing text alignment remains difficult. SPG resolves this by constructing a style noise vector from a third UNet branch whose self-attention Keys and Values are replaced by those extracted from a style image, then combining that style direction with the usual CFG direction so that sampling is jointly steered toward the target semantic and style distributions while preserving the original CFG contrast (Liang et al., 15 Aug 2025).

1. Problem setting and conceptual basis

In latent diffusion text-to-image generation, CFG combines a conditional prediction $\epsilon_\theta(x_t,t,c)$ and an unconditional prediction $\epsilon_\theta(x_t,t,\varnothing)$ to improve adherence to a text prompt. The basic tension addressed by SPG is that semantic control and style control are not equivalent operations. CFG is effective for text alignment, but style control from a reference image is not reliably achieved by simply injecting style features into both CFG branches.

The paper identifies a specific failure mode in training-free KV-injection baselines. When style Keys and Values are directly inserted into both the conditional and unconditional CFG branches, the contrast required by CFG is disrupted. The reported consequence is degraded semantic alignment together with weak or unstable stylization. SPG is motivated by preserving this contrast rather than modifying it indiscriminately.

The method’s core intuition is to isolate style as a directional deviation in noise space. Instead of entangling style with both CFG branches, SPG computes a third prediction: an unconditional UNet forward pass in which self-attention Keys and Values are replaced by those extracted from the style image, while the queries remain tied to the current generation trajectory. The difference between this style-conditioned unconditional prediction and the true unconditional prediction is treated as a style direction. This separation is what allows SPG to add style control without collapsing CFG’s semantic mechanism.

This design places SPG within a broader line of inference-time guidance methods for diffusion. Earlier style-guidance approaches also aimed to separate content from style at sampling time, including guidance based on style statistics from a reference image (Pan et al., 2022) and compositional guidance that decomposes prompts into conceptual elements with separate guidance terms (Shu et al., 2024). SPG differs in that it uses self-attention KV replacement to define a style-specific noise direction while explicitly retaining the original conditional–unconditional contrast of CFG.

2. Mathematical formulation

Let $x_0$ denote an image latent, $x_t$ the noisy latent at timestep $t$, and $\epsilon_\theta(x_t,t,\cdot)$ the UNet noise predictor. The conditional and unconditional predictions are

$$\epsilon_{\text{cond}}=\epsilon_\theta(x_t,t,c), \qquad \epsilon_{\text{uncond}}=\epsilon_\theta(x_t,t,\varnothing).$$

Standard CFG is written as

$$\epsilon_{\mathrm{CFG}}(x_t,t,c) = \epsilon_{\text{uncond}} + s_{\text{cfg}} \bigl(\epsilon_{\text{cond}}-\epsilon_{\text{uncond}}\bigr).$$

SPG introduces a style-conditioned unconditional prediction

$$\epsilon_{\text{style}} \triangleq \epsilon_{\theta}^{\text{uncond+KV}(s)}(x_t,t),$$

where self-attention Keys and Values are replaced by those extracted from the style reference image. The style direction is then

$$\Delta_{\text{style}} = \epsilon_{\text{style}}-\epsilon_{\text{uncond}}.$$

The combined SPG+CFG guidance becomes

$\epsilon_\theta(x_t,t,\varnothing)$0

This formulation is notable for its simplicity. The paper states that the guidance scales are constant across timesteps, with no normalization, cosine-weighting, or time-dependent schedule applied to $\epsilon_\theta(x_t,t,\varnothing)$1. In the reported setting, the constant values are $\epsilon_\theta(x_t,t,\varnothing)$2 and $\epsilon_\theta(x_t,t,\varnothing)$3.

SPG is inserted directly into the sampler by replacing the usual noise estimate with $\epsilon_\theta(x_t,t,\varnothing)$4. For a DDPM-style update,

$\epsilon_\theta(x_t,t,\varnothing)$5

For DDIM with $\epsilon_\theta(x_t,t,\varnothing)$6, the paper states that the deterministic update is used with $\epsilon_\theta(x_t,t,\varnothing)$7 plugged into the standard DDIM formula.

A central implication of this formulation is that SPG is sampling-time only. It does not require retraining, finetuning, or architectural modification of the UNet; it only changes how the denoising prediction is formed during inference (Liang et al., 15 Aug 2025).

3. Construction of the style signal

SPG specifies style using a single reference image. No fine-tuning, learned tokens, or text-only style prompts are required. The method first performs DDIM inversion of the reference image to obtain noisy style latents $\epsilon_\theta(x_t,t,\varnothing)$8 along the same schedule used for generation. At each timestep, the UNet is run on $\epsilon_\theta(x_t,t,\varnothing)$9 and self-attention Keys and Values are extracted from the decoder’s last six self-attention blocks, corresponding to layers 65–70 in the SDXL implementation.

These cached tensors, $x_0$0 and $x_0$1, constitute the style representation used during sampling. At each generation step, SPG computes three UNet predictions:

1. $x_0$2 with the model unchanged.
2. $x_0$3 with the model unchanged.
3. $x_0$4 with self-attention Keys and Values replaced by $x_0$5 in the selected layers, while queries remain from the current target pass.

The separation between style Keys/Values and current-step queries is essential. The paper presents this as the mechanism by which local texture and stylistic statistics are transferred while preserving the target image’s evolving structure.

An additional component is Adaptive Instance Normalization on latents. Before the UNet calls at timestep $x_0$6, the current generation latent $x_0$7 can be replaced by

$x_0$8

where $x_0$9 and $x_t$0 are per-channel mean and standard deviation. The paper reports that this was critical for color and tonal fidelity in ablations. In effect, the attention-based style signal handles higher-level stylistic structure, while AdaIN improves low-level color statistics and contrast.

This design differs from other training-free style-guidance mechanisms. Earlier diffusion guidance based on AdaIN-style feature statistics used gradients on a noise-free proxy $x_t$1 to match reference style features (Pan et al., 2022). SPG instead uses self-attention KV capture from DDIM-inverted reference latents and treats the resulting deviation from unconditional noise prediction as an explicit style direction.

4. Sampling algorithm, implementation, and integrations

A single SPG timestep takes as input the current latent $x_t$2, timestep $x_t$3, text embedding $x_t$4, cached style latents $x_t$5, cached style Keys and Values for the selected layers, and the scales $x_t$6 and $x_t$7. The per-step procedure is:

• optionally replace $x_t$8 by $x_t$9;
• compute $t$0;
• compute $t$1;
• compute $t$2 via unconditional forward pass with KV replacement;
• form $t$3;
• update $t$4 with the chosen sampler.

The default reported configuration uses SDXL as backbone, DDIM sampling with 50 steps and $t$5, self-attention injection in the last six decoder layers, and guidance scales $t$6, $t$7. Runtime is reported as approximately 30 seconds per 1024px SDXL stylized image on a single RTX 3090. The method is also shown to work on SD v1.5, where inference is reported at approximately 7 seconds per image on the same GPU.

Relative to vanilla CFG, which requires two UNet passes per step, SPG adds one extra pass for the style branch, giving approximately $t$8 compute over CFG-only sampling. Additional overhead comes from the one-time DDIM inversion of the style image and from storing $t$9 across timesteps. The paper argues that this memory cost is practical because only the last six layers are cached.

A practical feature of SPG is compatibility with controllable frameworks. With ControlNet, the same ControlNet residuals are fed to all three branches—unconditional, conditional, and style—so that structural control is shared. With IPAdapter, the conditional branch uses image conditioning in place of the text prompt, while the SPG branch remains unconditional with KV injection only, preserving disentanglement between content and style. This implementation strategy reflects the paper’s broader claim that SPG is not a replacement for structural or content conditioning, but an additive style-control mechanism layered on top of them (Liang et al., 15 Aug 2025).

5. Experimental findings and ablations

The primary experiments are conducted on SDXL, with additional results on SD v1.5 to demonstrate generality. Styles are specified by single reference images collected from the Internet, and prompts consist of nouns such as guitar, deer, chair, and balls, with five images generated per prompt. Text alignment is measured by CLIP ViT-B/32 cosine similarity between generated images and prompts, and style alignment by DINO ViT-B/8 cosine similarity between generated images and the style reference.

The baselines include training-free KV-injection methods such as StyleAligned, Attention Distillation, and VSP, as well as the authors’ own KV-injection baselines with and without AdaIN. The qualitative conclusion reported in the paper is that direct KV-injection methods frequently compromise text alignment or produce weak stylization. StyleAligned is described as introducing Q/K/V entanglement in both CFG branches and disrupting CFG contrast. VSP’s concatenation of Keys and Values is described as weakening both image quality and style alignment. Attention Distillation is said to optimize a KV loss that implicitly entangles style with both unconditional and conditional latents, again affecting semantics.

Quantitatively, the paper reports that exact numeric values are not provided, but the scatter plot places SPG in the Pareto-best region, achieving the best balance between CLIP text alignment and DINO style similarity among all tested methods. This is a comparative claim about joint performance rather than an isolated metric maximum.

The ablations are especially informative. Removing AdaIN substantially worsens color and tonal matching. Injecting all attention layers causes reconstruction of the style reference’s structure and harms semantics, whereas restricting injection to the last six decoder self-attention layers yields a better balance between local texture transfer and content preservation. The stylization effect is reported as stable for $\epsilon_\theta(x_t,t,\cdot)$0, with $\epsilon_\theta(x_t,t,\cdot)$1 recommended as default. Removing CFG harms semantic fidelity, which the paper presents as evidence that SPG is designed to complement, not replace, CFG.

The experiments therefore support a specific interpretation of SPG: the method’s performance depends less on maximizing style transfer strength than on preserving a clean decomposition between semantic guidance and style guidance. This concern with disentanglement is closely related to later work on preventing content leakage in visual style prompting, where negative visual query guidance is introduced to suppress unwanted reference content during self-attention-based transfer (Jeong et al., 8 Oct 2025).

6. Practical use, limitations, and position in the literature

The paper recommends SPG whenever a single style reference image is available and style fidelity matters. It explicitly contrasts this with CFG-only generation, which cannot reliably enforce style. Typical tuning guidance depends on style type. Painterly and texture-rich styles are associated with $\epsilon_\theta(x_t,t,\cdot)$2–$\epsilon_\theta(x_t,t,\cdot)$3 and strong reliance on AdaIN. Comic and line-art styles may require slightly larger $\epsilon_\theta(x_t,t,\cdot)$4, especially when combined with ControlNet line-art or edge maps to preserve structure. Photorealistic styles are said to benefit from lower $\epsilon_\theta(x_t,t,\cdot)$5 values, typically $\epsilon_\theta(x_t,t,\cdot)$6–$\epsilon_\theta(x_t,t,\cdot)$7, with high $\epsilon_\theta(x_t,t,\cdot)$8.

Several failure modes are identified. Excessively large $\epsilon_\theta(x_t,t,\cdot)$9 or injection into too many layers can produce semantic drift or structure copying from the reference. Weak stylization can result from subtle style textures, disabled AdaIN, or mismatched DDIM inversion schedules. Conflicts between desired content and the style prior can be mitigated by increasing $$\epsilon_{\text{cond}}=\epsilon_\theta(x_t,t,c), \qquad \epsilon_{\text{uncond}}=\epsilon_\theta(x_t,t,\varnothing).$$0, reducing $$\epsilon_{\text{cond}}=\epsilon_\theta(x_t,t,c), \qquad \epsilon_{\text{uncond}}=\epsilon_\theta(x_t,t,\varnothing).$$1, or choosing a reference whose global color and texture are more compatible with the target content.

The limitations section is concise but technically significant. SPG is currently unsupported on distilled models that do not use CFG-like dual-branch sampling, with FLUX given as the example. Some styles with strong geometry priors can still bias structure. The paper also notes that no time-dependent scheduling, normalization, or cosine reweighting was explored, and presents per-layer or per-head weighting as a possible future direction.

Within the broader literature, SPG occupies a specific methodological niche. It is training-free and sampling-time only, like earlier arbitrary style guidance methods (Pan et al., 2022), yet unlike those methods it does not rely on backpropagated style gradients through external feature networks. It also differs from compositional guidance systems such as DreamWalk, which decompose prompt concepts and assign spatiotemporal guidance scale functions to each component (Shu et al., 2024). SPG’s contribution is narrower but more targeted: it defines style as a directional deviation in diffusion noise space derived from a style-conditioned unconditional branch, then adds that direction to standard CFG without altering CFG’s semantic contrast.

In that sense, SPG is best understood not as a general theory of style control, but as a concrete inference-time mechanism for isolating and injecting style in latent diffusion samplers. Its reported strength lies in the combination of mathematical simplicity, absence of training, compatibility with existing control modules, and an empirically favorable balance between textual alignment and style fidelity (Liang et al., 15 Aug 2025).