Controllable Stylized Distillation

• Controllable stylized distillation is defined as modifying generative distillation objectives to incorporate style as an explicit control signal, enabling joint optimization of content and style.
• It leverages techniques such as score mixing, self-attention manipulation, and dual encoder architectures to ensure style is injected without compromising content or geometry.
• Empirical strategies including dynamic scheduling and tailored guidance mechanisms effectively balance style strength and content preservation, addressing challenges like geometry corruption and content leakage.

Searching arXiv for the primary paper and closely related controllable stylization/distillation work. Controllable stylized distillation denotes a set of generative strategies in which style is introduced as an explicit control variable inside a distillation, guidance, or representation-steering objective, rather than being left to prompt engineering alone. In the literature represented here, the idea is instantiated most directly in text-to-3D as stylized score distillation, where content and style are jointly optimized from a text prompt and a style reference image, and it is subsequently reformulated for 3D Gaussian Splatting, unsupervised image style transfer, one-step diffusion control, and transferable control modules for distilled diffusion models (Kompanowski et al., 2024, Yang et al., 11 Aug 2025, Yang et al., 2 Aug 2025, Luo et al., 9 Mar 2025, Gandikota et al., 13 Mar 2025).

1. Definition and conceptual scope

In its narrowest sense, controllable stylized distillation is the modification of a generative distillation objective so that style enters as a first-class conditioning signal with an explicit strength parameter. In "Dream-in-Style" (Kompanowski et al., 2024), this is realized by combining two denoising processes on the same pretrained diffusion backbone: an original text-conditioned branch for content, and a modified branch whose self-attention key/value features are manipulated to inject style from a reference image. The resulting optimization updates a NeRF-style volumetric 3D representation so that rendered views match the text prompt while following the reference style.

In a broader sense, the term covers several related design patterns. "FantasyStyle" (Yang et al., 11 Aug 2025) treats controllable stylized distillation as a redesigned diffusion objective for 3DGS style transfer that removes the reconstruction term of SDS/DDS and introduces negative guidance to suppress content leakage from the style image. "StyDeco" (Yang et al., 2 Aug 2025) uses the phrase in a looser but structurally related way: stylistic priors are distilled into pseudo-paired data and a lighter generator, while content and style are separated by domain-specific text encoders. "Adding Additional Control to One-Step Diffusion with Joint Distribution Matching" (Luo et al., 9 Mar 2025) generalizes the same intuition beyond style, by matching image-condition joint distributions so that a one-step student can acquire controls unknown to the teacher. "Distilling Diversity and Control in Diffusion Models" (Gandikota et al., 13 Mar 2025) shows that control mechanisms such as Concept Sliders and LoRAs can transfer between base and distilled models without retraining, framing control itself as a preserved structure under distillation.

This suggests that controllable stylized distillation is best understood not as a single algorithm but as a family of methods with a common objective: separating content-preserving generative priors from style-bearing control signals, then recombining them through a controllable operator.

2. Score-space formulations

The canonical formulation is Dream-in-Style’s stylized score distillation. Standard SDS optimizes parameters $\theta$ so that a rendering $x = g(\theta)$ lies on the text-conditional image manifold of a pretrained diffusion model through the gradient

$$\nabla_{\theta} \mathcal{L}_{SDS} = \mathbb{E}_{t, \epsilon} \left[ \omega(t) \left( \epsilon_{\phi}(z_t \mid y)- \epsilon \right) \frac{\partial x}{\partial \theta} \right],$$

with $z_t = \alpha_t x + \sigma_t \epsilon$. Dream-in-Style replaces the single score with a convex combination of an original score $\epsilon_\phi(z_t \mid y)$ and a style-injected score $\hat{\epsilon}_\phi(z_t \mid y, s)$, controlled by a style ratio $\lambda \in [0,1]$:

$$\nabla_{\theta} \mathcal{L}_{SSD} = \mathbb{E}_{t, \epsilon} \left[ \omega(t) \left( (1 - \lambda) \epsilon_{\phi}(z_t \mid y) + \lambda \hat{\epsilon}_{\phi}(z_t \mid y, s) - \epsilon \right) \frac{\partial x}{\partial \theta} \right].$$

The corresponding target distribution is defined by

$$\log p_\phi(z_t \mid y, s) = (1-\lambda) \log p_\phi(z_t \mid y) + \lambda\log \hat{p}_\phi(z_t \mid y, s),$$

so the stylized objective remains formally close to SDS while making style strength explicit (Kompanowski et al., 2024).

Dream-in-Style also adapts Noise-Free Score Distillation. There, the score is decomposed into a domain direction $\delta_D$ and a conditioning direction $x = g(\theta)$0, and both the original and stylized branches are mixed:

$x = g(\theta)$1

Because both domain and conditioning directions are interpolated, the method gives a more interpretable decomposition of content-preserving and style-bearing forces (Kompanowski et al., 2024).

FantasyStyle adopts a different score-space intervention for 3DGS style transfer. Its Controllable Stylized Distillation builds on DDS but removes the reconstruction term, which the paper argues smooths out high-frequency style details and slows optimization when geometry is fixed and only color is optimized. The target branch uses style-plus-negative guidance, while the source branch uses standard text CFG:

$x = g(\theta)$2

$x = g(\theta)$3

and the gradient becomes

$x = g(\theta)$4

This formulation is still a distillation objective, but style control is now expressed through asymmetric branch design and negative guidance rather than through a direct convex mixture of content and stylized scores (Yang et al., 11 Aug 2025).

A more general abstraction appears in Joint Distribution Matching. There the distillation target is a joint distribution $x = g(\theta)$5, and the student minimizes a reverse-KL upper bound whose gradient decomposes into a condition-learning term and a fidelity-learning term:

$x = g(\theta)$6

This does not specialize to style, but it supplies a general mathematical template for controllable distillation in which style is one possible condition variable $x = g(\theta)$7 (Luo et al., 9 Mar 2025).

3. Mechanisms for injecting and separating style

Across the literature, the technical problem is not merely to add style, but to add it without corrupting content, geometry, or truth-bearing structure. One recurring mechanism is self-attention manipulation. Dream-in-Style’s modified sibling is not a separately trained model; it shares weights with the original pretrained diffusion network and injects style by swapping or sharing self-attention key/value features from a parallel style process. Formally,

$x = g(\theta)$8

where the content path uses queries from the content branch and keys/values from the style branch. This preserves semantic content from text while introducing style patterns through the attention stream (Kompanowski et al., 2024).

Attention-level separation is developed in a different way by "StyleAdapter" (Wang et al., 2023). Rather than distilling scores, it separates content and style into two parallel cross-attention paths: a frozen text path and a learnable style path. Their outputs are fused as

$x = g(\theta)$9

The paper further suppresses semantics in style references by shuffling positional embeddings, removing the CLIP class token, and using multiple style references. Although this is not itself a distillation method, it provides a direct architectural recipe for style-content separation that later controllable distillation methods can reuse (Wang et al., 2023).

FantasyStyle attacks a different failure mode: content leakage from the style image. Instead of manually writing negative prompts, it uses IPAdapter-Instruct to split the style image into style features $$\nabla_{\theta} \mathcal{L}_{SDS} = \mathbb{E}_{t, \epsilon} \left[ \omega(t) \left( \epsilon_{\phi}(z_t \mid y)- \epsilon \right) \frac{\partial x}{\partial \theta} \right],$$0 and content features $$\nabla_{\theta} \mathcal{L}_{SDS} = \mathbb{E}_{t, \epsilon} \left[ \omega(t) \left( \epsilon_{\phi}(z_t \mid y)- \epsilon \right) \frac{\partial x}{\partial \theta} \right],$$1, then replaces the null condition in CFG with the content features of the style image:

$$\nabla_{\theta} \mathcal{L}_{SDS} = \mathbb{E}_{t, \epsilon} \left[ \omega(t) \left( \epsilon_{\phi}(z_t \mid y)- \epsilon \right) \frac{\partial x}{\partial \theta} \right],$$2

The model is therefore encouraged to move away from the style image’s content and toward its style (Yang et al., 11 Aug 2025).

A third line of work separates style and content in representation space rather than attention space. StyDeco introduces two domain-specific text encoders through LoRA adaptation,

$$\nabla_{\theta} \mathcal{L}_{SDS} = \mathbb{E}_{t, \epsilon} \left[ \omega(t) \left( \epsilon_{\phi}(z_t \mid y)- \epsilon \right) \frac{\partial x}{\partial \theta} \right],$$3

and uses them in a cycle-consistent forward stylization and backward de-stylization loop. In this paper, the abbreviation CSD denotes Contrastive Semantic Decoupling rather than Controllable Stylized Distillation. The effect is a two-cluster semantic space in which one embedding predominantly encodes content and the other predominantly encodes target style (Yang et al., 2 Aug 2025).

A related but non-visual analogy appears in StyliTruth, which identifies latent coupling between style and truth directions in key attention heads of an LLM and uses orthogonal deflation to separate them. This is not image generation, but it sharpens the general principle that controllability improves when style-relevant and task-critical subspaces are explicitly disentangled (Shen et al., 6 Aug 2025).

4. Representative instantiations across domains

The main instantiations differ by generative substrate, but they share a common structure: a pretrained prior supplies fidelity, a separate mechanism supplies style, and a control parameter or structured conditioning determines their balance.

Work	Setting	Main control mechanism
Dream-in-Style	Text-to-3D with NeRF	Score mixing with style ratio $$\nabla_{\theta} \mathcal{L}_{SDS} = \mathbb{E}_{t, \epsilon} \left[ \omega(t) \left( \epsilon_{\phi}(z_t \mid y)- \epsilon \right) \frac{\partial x}{\partial \theta} \right],$$4
FantasyStyle	3DGS style transfer	Negative guidance and reconstruction-term removal
StyDeco	Unsupervised image style transfer	Prior-Guided Data Distillation plus dual text encoders
JDM	One-step diffusion	Joint distribution matching over $$\nabla_{\theta} \mathcal{L}_{SDS} = \mathbb{E}_{t, \epsilon} \left[ \omega(t) \left( \epsilon_{\phi}(z_t \mid y)- \epsilon \right) \frac{\partial x}{\partial \theta} \right],$$5
Distilling Diversity and Control	Distilled SDXL	Transfer of LoRAs, Concept Sliders, and hybrid first-step inference
Attention Distillation	Diffusion editing and sampling	Latent optimization to match ideal attention outputs

Dream-in-Style operates on a NeRF-style neural radiance field implemented via threestudio and NerfAcc, with random camera sampling and differentiable rendering. The method is SDS-compatible and adapts plain SDS, NFSD, and VSD by substituting the usual score with the stylized mixture. Prompt augmentation in the stylized branch concatenates the content prompt with a BLIP2 caption of the style image, and runtime is reported as approximately 1.5 hours per object for SDS/NFSD and approximately 2.5 hours for VSD on an RTX 4090 (Kompanowski et al., 2024).

FantasyStyle reformulates the problem for reconstructed 3DGS scenes. Geometry is fixed and only color-related Gaussian parameters are optimized. Its second core component, Multi-View Frequency Consistency, applies 3D frequency filtering to multi-view noisy latents, retaining high-frequency texture information while modulating low-frequency components to reduce cross-view style conflicts. The target branch of CSD operates on MVFC-processed latents before SDXL denoising, so stylization and multi-view consistency are coupled at the latent level rather than enforced by a separate VGG-style regularizer (Yang et al., 11 Aug 2025).

StyDeco shifts to unsupervised 2D style transfer. Prior-Guided Data Distillation synthesizes pseudo-paired data with a frozen editor such as InstructPix2Pix, and the student is a one-step diffusion generator trained jointly with two LoRA-adapted text encoders. The framework also supports a de-stylization process by feeding a content-domain embedding into the same generator, making control bidirectional rather than purely stylizing (Yang et al., 2 Aug 2025).

At the level of accelerated diffusion, JDM and "Distilling Diversity and Control in Diffusion Models" identify two distinct kinds of controllability. JDM makes new controls available to a one-step student without modifying the teacher by decoupling fidelity learning from condition learning. Distilling Diversity and Control shows that control modules trained on SDXL-Base, including Concept Sliders and LoRA-based style controls, can be used on distilled models and vice versa without retraining, and that a hybrid inference scheme using the base model for the first critical timestep can recover or exceed base-model diversity while preserving nearly distilled-model efficiency (Luo et al., 9 Mar 2025, Gandikota et al., 13 Mar 2025).

Attention Distillation occupies an intermediate position between editing and distillation. It defines an ideal attention output obtained by pairing current queries with reference keys and values, then optimizes the latent so that the current attention output matches that ideal:

$$\nabla_{\theta} \mathcal{L}_{SDS} = \mathbb{E}_{t, \epsilon} \left[ \omega(t) \left( \epsilon_{\phi}(z_t \mid y)- \epsilon \right) \frac{\partial x}{\partial \theta} \right],$$6

A content-preserving term on query features,

$$\nabla_{\theta} \mathcal{L}_{SDS} = \mathbb{E}_{t, \epsilon} \left[ \omega(t) \left( \epsilon_{\phi}(z_t \mid y)- \epsilon \right) \frac{\partial x}{\partial \theta} \right],$$7

provides an explicit content-style trade-off, and the same loss can be integrated into DDIM sampling as an energy-like guidance term (Zhou et al., 27 Feb 2025).

5. Control parameters, schedules, and empirical trade-offs

The most explicit control variable in the literature is Dream-in-Style’s style ratio $$\nabla_{\theta} \mathcal{L}_{SDS} = \mathbb{E}_{t, \epsilon} \left[ \omega(t) \left( \epsilon_{\phi}(z_t \mid y)- \epsilon \right) \frac{\partial x}{\partial \theta} \right],$$8. When $$\nabla_{\theta} \mathcal{L}_{SDS} = \mathbb{E}_{t, \epsilon} \left[ \omega(t) \left( \epsilon_{\phi}(z_t \mid y)- \epsilon \right) \frac{\partial x}{\partial \theta} \right],$$9, optimization behaves like standard text-to-3D; when $z_t = \alpha_t x + \sigma_t \epsilon$0, stylization is strong but geometry can be corrupted. The paper therefore emphasizes dynamic scheduling. A square-root schedule,

$z_t = \alpha_t x + \sigma_t \epsilon$1

and a quadratic schedule,

$z_t = \alpha_t x + \sigma_t \epsilon$2

are used so that early iterations prioritize 3D structure and later iterations prioritize stylization. The reported best general choice is the sqrt schedule with $z_t = \alpha_t x + \sigma_t \epsilon$3, while a quad schedule with $z_t = \alpha_t x + \sigma_t \epsilon$4 is preferable for very abstract style images with no clear foreground object (Kompanowski et al., 2024).

FantasyStyle’s primary control is not a mixture coefficient but the guidance design itself. Because the reconstruction term is removed, stylization strength is largely driven by CFG scale $z_t = \alpha_t x + \sigma_t \epsilon$5, and the paper reports that CSD is less sensitive to $z_t = \alpha_t x + \sigma_t \epsilon$6 than SDS/DDS. Here controllability is also global: the paper explicitly notes that finer, region-wise control of stylization strength is not explored (Yang et al., 11 Aug 2025).

In StyDeco, control is exerted through the choice of encoder branch. Feeding $z_t = \alpha_t x + \sigma_t \epsilon$7 stylizes, while feeding $z_t = \alpha_t x + \sigma_t \epsilon$8 de-stylizes. The paper does not report an explicit scalar style-strength parameter, but it notes that interpolation between the two embeddings is a plausible control mechanism:

$z_t = \alpha_t x + \sigma_t \epsilon$9

This is presented as a plausible implication rather than an evaluated result (Yang et al., 2 Aug 2025).

Control strength can also be attached to inference-time modules rather than training-time losses. In Distilling Diversity and Control, a slider value $\epsilon_\phi(z_t \mid y)$0 scales a Concept Slider direction in LoRA form, while the diversity-control trade-off is handled by deciding whether the first timestep is taken with the base model or the distilled model. The paper argues that initial diffusion timesteps disproportionately determine output diversity, while later steps primarily refine details (Gandikota et al., 13 Mar 2025).

Empirically, the trade-off between stylization and preservation is measured differently across tasks. Dream-in-Style extends GPTEval3D with style alignment and reports overall Elo of approximately 1140, compared with a baseline Elo of 1000 for style-in-prompt, with the highest category score in style alignment as well as strong geometry and text alignment. FantasyStyle reports ArtFID $\epsilon_\phi(z_t \mid y)$1, $\epsilon_\phi(z_t \mid y)$2 $\epsilon_\phi(z_t \mid y)$3, $\epsilon_\phi(z_t \mid y)$4 $\epsilon_\phi(z_t \mid y)$5, and improved multi-view consistency metrics relative to StyleGaussian and SGSST. StyDeco reports, on Van Gogh, FID $\epsilon_\phi(z_t \mid y)$6, SSIM $\epsilon_\phi(z_t \mid y)$7, LPIPS $\epsilon_\phi(z_t \mid y)$8, and CLIP-ae $\epsilon_\phi(z_t \mid y)$9, emphasizing strong structural preservation with competitive style fidelity (Kompanowski et al., 2024, Yang et al., 11 Aug 2025, Yang et al., 2 Aug 2025).

These results support a recurring empirical claim: direct maximization of style signal is insufficient. The best-performing systems either mix content and style scores, explicitly remove style-image content, or decouple content and style representations before control is applied.

6. Limitations, misconceptions, and directions

A common misconception is that stronger style guidance monotonically improves results. The evidence is the opposite. Dream-in-Style reports that $\hat{\epsilon}_\phi(z_t \mid y, s)$0 achieves strong style similarity but severely corrupted or ambiguous geometry, and that starting stylization from the beginning with a high constant $\hat{\epsilon}_\phi(z_t \mid y, s)$1 degrades geometry (Kompanowski et al., 2024). FantasyStyle likewise argues that VGG-based style transfer often causes content leakage and excessive stylization, while SDS/DDS reconstruction terms blur brushstroke details in color-only 3DGS transfer (Yang et al., 11 Aug 2025). In both cases, control is beneficial only when style is structurally constrained.

Another misconception is that distilled models necessarily lose controllability. Distilling Diversity and Control finds that control modules such as Concept Sliders and LoRAs trained on the base model can be used on distilled models and vice versa without retraining, with similar strength of control. What is more vulnerable than control is sample diversity: the paper attributes diversity collapse to the earliest timesteps and proposes a hybrid inference scheme that restores or exceeds base-level diversity while remaining nearly as efficient as distilled inference (Gandikota et al., 13 Mar 2025).

Several limitations recur across formulations. Dream-in-Style remains subject to the Janus problem and depends on SDS-family optimization; adapting it to non-SDS 3D generation is explicitly left for future work. FantasyStyle is not real-time, depends on SDXL and IP-Adapter-Instruct, and offers only global control via $\hat{\epsilon}_\phi(z_t \mid y, s)$2. StyDeco depends heavily on the frozen prior used for pseudo-pair generation and evaluates only three artistic styles. JDM requires accurate modeling of $\hat{\epsilon}_\phi(z_t \mid y, s)$3 and is theoretically derived under a discrete-condition assumption. Attention Distillation incurs backward-pass overhead and can fail when content and style are semantically very different (Kompanowski et al., 2024, Yang et al., 11 Aug 2025, Yang et al., 2 Aug 2025, Luo et al., 9 Mar 2025, Zhou et al., 27 Feb 2025).

The trajectory suggested by these papers is consistent. Future work is repeatedly framed in terms of multi-style control, spatially varying control, lighter or task-specific diffusion backbones, and more explicit separation of controllable factors. This suggests that controllable stylized distillation is evolving toward a general mixture-of-experts or disentangled-subspace view of generation: one component preserves fidelity, geometry, or truth; another carries stylization; and the principal research problem is to make their interaction continuous, localizable, and stable under acceleration and distillation.