---
title: Control-Side Score Distillation
url: https://www.emergentmind.com/topics/control-side-score-distillation-csd
type: topic
---

# Control-Side Score Distillation

Searching arXiv for the cited CSD-related papers and adjacent score-distillation work.
Control-side score distillation is not a standardized method name in the recent diffusion-based synthesis literature. A plausible unifying interpretation is to treat it as an *Editor’s term* for methods that modify the score-distillation update, the aggregation of score signals, or the gradient path through structured control variables, while keeping a pre-trained diffusion prior as the principal source of supervision. Under that reading, the literature groups together several distinct mechanisms: particle interaction across samples, canonical-and-warping factorization, classifier-side guidance, cascaded multi-resolution supervision, reward-weighted noise sampling, and DDIM-consistent noise control. At the same time, the acronym **CSD** is reused for several different expansions, so the term is inherently polysemous rather than canonical [2307.04787][2312.03795][2310.19415][2311.09571][2503.09601][2405.15891].

## 1. Terminology and scope

The first technical requirement is disambiguation. In the supplied literature, **CSD** does not denote a single formulation.

| Usage in the literature | Meaning of CSD | Core control locus |
|---|---|---|
| "Collaborative Score Distillation" [2307.04787] | multi-sample SDS via SVGD | kernel-weighted score sharing and repulsion |
| "Canonical Score Distillation" [2312.03795] | canonical-field and warping-field distillation for 4D | explicit gradients through canonical generation and warping refinement |
| "Classifier Score Distillation" [2310.19415] | CFG-difference-only distillation | implicit classifier gradient |
| "Cascaded Score Distillation" [2311.09571] | multi-stage diffusion supervision | stage-wise multi-resolution score aggregation |
| "Concrete Score Distillation" [2509.25837] | LLM distillation by concrete score matching | relative logit differences across vocabulary pairs |

This terminological collision matters because each formulation changes a different part of the SDS pipeline. A plausible implication is that “control-side” is best reserved for the **location of intervention**—the score-distillation layer, the aggregation rule, or the structured conditioning path—rather than for any single acronym expansion.

## 2. SDS as the common substrate

Most of the visual formulations in this family inherit the standard SDS setup in which a differentiable generator or renderer \(g(\theta)\) is optimized under a frozen text-conditioned diffusion prior. In that baseline, the image or rendering is \(x_0 = g(\theta)\), the noisy sample is \(x_t = \alpha_t x_0 + \sigma_t \epsilon_t\), and the SDS gradient is

\[
\nabla_\theta L_{SDS}(x_0 = g(\theta)) = \mathbb{E}_{t,\epsilon_t} \left[ w(t) (\epsilon_\phi(x_t, y, t) - \epsilon_t) \frac{\partial x_0}{\partial \theta} \right].
\]

RewardSDS states the key limitation of this baseline directly: standard SDS averages gradients over noise samples \(\epsilon_t\) and timesteps \(t\) **uniformly**, so every noise draw at a given timestep contributes equally even though some draws are more aligned than others [2503.09601]. The same paper also characterizes VSD as performing this procedure “in a variational/particle way using an auxiliary LoRA-tuned diffusion” [2503.09601].

Under the control-side reading, the unifying move is not to replace the diffusion prior, but to change **how** that prior is used. This suggests a taxonomy in which the intervention can act on one of four axes: the interaction among samples, the factorization of gradients through structured latent controls, the composition of the score itself, or the Monte Carlo estimator over noises and timesteps.

## 3. Multi-sample and structured-control formulations

"Collaborative Score Distillation" generalizes SDS from one optimized sample to a set of jointly optimized samples, treating multiple images as particles in a Stein Variational Gradient Descent update. For particles \(\{\theta_i\}_{i=1}^N\), rendered images \(\mathbf{x}^{(i)} = g(\theta_i)\), and noisy images \(\mathbf{x}_t^{(i)}\), its core gradient is

\[
\nabla_{\theta_i} \mathcal{L}_{\tt{CSD}\big(\theta_i\big) = \frac{w(t)}{N}\sum_{j=1}^N \left( k(\mathbf{x}_t^{(j)}, \mathbf{x}_t^{(i)}) \big(\boldsymbol{\epsilon}_\phi^{\omega}(\mathbf{x}_t^{(j)};y,t) - \boldsymbol{\epsilon} \big) + \nabla_{\mathbf{x}_t^{(j)}} k(\mathbf{x}_t^{(j)}, \mathbf{x}_t^{(i)}) \right)\frac{\partial \mathbf{x}^{(i)}}{\partial \theta_i}.
\]

The two decisive terms are the **kernel-weighted averaging** of per-particle scores and the **repulsive gradient of the kernel**. If \(N = 1\), \(k(\mathbf{x},\mathbf{x}) = 1\), and \(\nabla k(\mathbf{x},\mathbf{x}) = 0\), the update reduces to SDS exactly. The mechanism is therefore an exact multi-particle generalization of SDS rather than a separate prior. Similar images exchange score information strongly, while the repulsive term prevents collapse [2307.04787].

"Canonical Score Distillation" changes a different part of the pipeline. AnimatableDreamer factorizes a 4D articulated model into a **canonical model** and a **warping / motion field**, then backpropagates diffusion supervision through both paths. Its core gradient is

\[
\nabla_\phi \mathcal{L}_{\text{CSD}}
= \mathbb{E}_{t,\,T,\,\mathbf{p},\,\epsilon}
\Big[
w(T)\big(\epsilon_\theta(I_{t,T};y,\mathbf{p},T)-\epsilon\big)
\frac{\partial \mathcal{R}_{\phi_*}(\mathbf{X}^*)}{\partial \mathbf{X}^*}
\frac{\partial W_{\phi_w}(\mathbf{X}^t)}{\partial \phi_w}
\Big].
\]

The paper explicitly names the two inner Jacobian factors **canonical generation** and **warping refinement**. In that formulation, CSD “lifts” 2D diffusion for “temporal consistent 4D generation,” while also refining bones and skinning so that articulated poses remain plausible under multi-view supervision [2312.03795].

Taken together, these two formulations show two distinct control-side strategies. Collaborative CSD imposes interaction **between samples**. Canonical CSD imposes structure **inside the generator**, using diffusion gradients to refine not only appearance-bearing variables but also bones, skinning weights, and warping fields.

## 4. Classifier-side guidance and cascaded supervision

"Classifier Score Distillation" reinterprets the practical success of SDS with classifier-free guidance. The key observation is that the CFG difference

\[
\epsilon_\phi(\mathbf{x}_t; y,t) - \epsilon_\phi(\mathbf{x}_t; t)
\]

is proportional to the gradient of an implicit classifier for \(q(y|\mathbf{x}_t)\). CSD therefore discards the generative-prior term and retains only the classifier score, yielding

\[
\nabla_\theta \mathcal{L}_\text{CSD}
=
\mathbb{E}_{t,\boldsymbol{\epsilon},\mathbf{c}}
\left[
w(t)\,
\big(
\epsilon_\phi(\mathbf{x}_t; y,t) - \epsilon_\phi(\mathbf{x}_t; t)
\big)
\frac{\partial \mathbf{x}}{\partial \theta}
\right].
\]

The resulting view is explicitly classifier-side: diffusion is used as an implicit text-image classifier rather than as a conditional density to be matched by KL. Negative prompts become optimization against two implicit classifiers, and text-guided editing becomes a weighted difference between a target classifier score and an edit-attribute classifier score [2310.19415].

"Cascaded Score Distillation" alters the score-distillation side along a scale axis. 3D Paintbrush uses a cascaded text-to-image diffusion model \(\{\phi^1,\phi^2,\ldots,\phi^N\}\), with stage 1 providing global semantic and positional supervision and later super-resolution stages providing high-frequency detail. For stage \(i>1\), the image-space gradient is

\[
\nabla_{x^i} \mathcal{L}_{CSD^i}(\phi^i, x^i, x^{i-1}, y)
=
w(t)\left(
\epsilon_{\phi^i}(z^i_t, t, z^{i-1}_s, s, y) - \epsilon^i
\right),
\]

and the full gradient combines stage-wise terms with user-set weights \(\lambda^i\). In 3D Paintbrush this supervision is applied jointly to a localization map, a local texture map, and a background texture map, so the diffusion prior supervises not only appearance but also **where** the edit should occur [2311.09571].

These two formulations expose complementary control principles. Classifier CSD isolates the text-discriminative part of the score. Cascaded CSD structures supervision across multiple resolutions and multiple rendered branches.

## 5. Reward weighting and DDIM-consistent noise control

RewardSDS introduces control at the level of the Monte Carlo estimator over noises. For each rendered image \(x_0 = g(\theta)\), the method samples multiple noises at one timestep, denoises each noisy image, scores the denoised images with a reward model \(R\), converts the reward scores to weights, and forms a reward-weighted SDS gradient:

\[
\nabla_\theta L_{R\text{-}SDS}(x_0 = g(\theta))
=
\mathbb{E}_{t}
\left[
\frac{1}{N}\sum_{i=1}^N
w^{(i)}
\left[
w(t)
(\epsilon_\phi(x_t^{(i)}, y, t) - \epsilon_t^{(i)})
\frac{\partial x_0}{\partial \theta}
\right]
\right].
\]

The important conceptual statement is explicit: control is applied on the **sampling / weighting of gradient contributions** on the “SDS side,” not on the denoising network architecture or parameters. The best-performing weighting scheme in the reported ablation assigns \(w^{(i)} = 0.9\) for the top 2 reward samples, \(w^{(i)} = -0.1\) for the bottom 2, and \(w^{(i)} = 0\) for all others [2503.09601].

"Score Distillation via Reparametrized DDIM" moves control to the noise term itself. It argues that SDS is equivalent to a DDIM-like update in the \(x_0\) space, but with freshly sampled i.i.d. noise at every step rather than a noise term consistent with the DDIM trajectory. The proposed SDI gradient is

\[
\nabla_\psi\mathcal{L}_{\mathrm{SDI}}
=
\sigma(t)
\left[
\epsilon_\theta^t\bigl(x(t), y\bigr) - \kappa^t_y(g(\psi, c))
\right]
\frac{\partial g}{\partial \psi},
\]

where \(\kappa^t_y(x_0)\) is approximated by conditional DDIM inversion plus a small Gaussian jitter. The paper’s central claim is that the excessive variance of SDS originates from replacing DDIM-consistent noise by i.i.d. noise, and that inversion-based noise recovery makes score distillation much closer to DDIM sampling [2405.15891].

A plausible implication is that control-side score distillation includes not only changes to *what* score is distilled, but also changes to *which stochastic realization* is allowed to define that score.

## 6. Applications, empirical behavior, and unresolved terminology

The empirical record across these papers is broad. Collaborative CSD is reported on panorama image editing, video editing, 3D scene editing, and a DreamFusion-style text-to-3D ablation. In video editing, CSD-Edit reports **higher CLIP directional (0.320 vs 0.314 / 0.230), higher CLIP image consistency (0.957 vs 0.948 / 0.949), and lower LPIPS (0.236 vs 0.267 / 0.283)** than FateZero and Pix2Video. In 3D scene editing, it slightly improves all three reported metrics over IN2N: **CLIP dir: 0.239 vs 0.230; CLIP img: 0.995 vs 0.994; LPIPS: 0.043 vs 0.048**. In the DreamFusion-style ablation it improves **FID (247.1 vs 259.4 for SDS baseline)** and is described as reducing the Janus problem [2307.04787].

Canonical CSD is evaluated on both reconstruction and text-guided generation from monocular video. In monocular reconstruction, the ablation without \(\mathcal{L}_{\text{CSD}}\) degrades Cat-Coco from **CD 3.65 / F@2% 63.3** to **CD 8.34 / F@2% 32.6**. In text-to-4D generation, the full method reports **CLIP: 38.2**, **CLIP-T: 96.6**, **R-Precision@10: 87.5**, and **GPT Eval3D Elo: 1098**, with weaker results when \(\mathcal{L}_{\text{bone/skel}}\), \(\mathcal{L}_{\text{skel}}\), or \(\mathcal{L}_{\text{bone}}\) are removed [2312.03795].

Classifier CSD reports a text-to-3D **CLIP Score** of **78.6**, compared with **67.5** for DreamFusion and **74.9** for Magic3D, and a user preference of **59.4%** over DreamFusion, Magic3D, and other baselines. In texture synthesis, user preference is **57.7%** in favor of CSD, and the paper reports runtime of **~1 hour on a single A800 GPU** for mesh refinement, versus **~8 hours** for ProlificDreamer under similar conditions [2310.19415].

RewardSDS shows that reward weighting can be layered on top of existing SDS pipelines without changing the diffusion prior. With MVDream + NeRF, the reported changes are **CLIP: 26.19 → 27.12**, **Aesthetic: 5.83 → 5.97**, **LLM-G: 5.86 → 6.07**, **Alignment: 3.51 → 4.21**, and **Realism: 3.14 → 3.79**. With MVDream + 3D Gaussians, the reported changes are **CLIP: 24.71 → 25.24**, **Aesthetic: 5.73 → 5.79**, **LLM-G: 4.90 → 5.31**, **User MOS alignment: 2.74 → 4.11**, and **Realism: 2.28 → 3.13** [2503.09601].

Reparametrized DDIM provides a different empirical profile: it does not add multi-view interaction or reward models, but controls the noise process itself. The paper reports **CLIP Score: SDI ~33.47 vs SDS ~29.81** and **Divergence rate: SDI ~4.7% vs SDS ~18.6%**, alongside better CLIP IQA and lower instability [2405.15891].

These gains come with recurring limitations. Collaborative CSD inherits **all pairwise kernel evaluations** in the batch, i.e. \(O(N^2)\), and LPIPS-based kernels are more expensive than \(\ell_2\)-based kernels [2307.04787]. Canonical CSD requires rendering multiple views and backpropagating through camera, warping, and canonical field, which the paper identifies as **high memory and compute** [2312.03795]. Classifier CSD explicitly notes that direct application to pure 2D image optimization produces artifacts, even though the same mechanism works well in multi-view 3D [2310.19415]. RewardSDS adds repeated denoising and reward-model evaluation, with runtime increasing from **~45s per image** for baseline SDS to **~2227s per image** for a large-scale configuration [2503.09601]. SDI requires inversion steps and careful guidance schedules, which increases VRAM and compute relative to SDS even though it remains lighter than heavy multi-stage baselines [2405.15891].

One final ambiguity should be kept explicit. In a separate line of work outside visual diffusion, "Concrete Score Distillation" uses the same acronym **CSD** for a discrete score-matching objective in autoregressive LLM distillation. There, CSD aligns **relative logit differences across all vocabulary pairs between student and teacher with flexible weighting**, and the object of control is not a diffusion-guided renderer but a token-level KD objective [2509.25837]. This separate usage reinforces the broader point: **CSD is an overloaded acronym, whereas control-side score distillation is best treated as a description of where the intervention occurs in a score-distillation pipeline rather than as a single, settled method name**.

Source: https://www.emergentmind.com/topics/control-side-score-distillation-csd