Control-Side Score Distillation

• Control-side score distillation is a framework that modifies standard SDS pipelines by altering gradient aggregation and structured control variables.
• It integrates diverse mechanisms such as multi-sample interaction, canonical warping for 4D consistency, and classifier-side guidance.
• These approaches yield enhanced performance in image, video, and 3D editing while introducing trade-offs in computation and memory usage.

Searching arXiv for the cited CSD-related papers and adjacent score-distillation work. Control-side score distillation is not a standardized method name in the recent diffusion-based synthesis literature. A plausible unifying interpretation is to treat it as an Editor’s term for methods that modify the score-distillation update, the aggregation of score signals, or the gradient path through structured control variables, while keeping a pre-trained diffusion prior as the principal source of supervision. Under that reading, the literature groups together several distinct mechanisms: particle interaction across samples, canonical-and-warping factorization, classifier-side guidance, cascaded multi-resolution supervision, reward-weighted noise sampling, and DDIM-consistent noise control. At the same time, the acronym CSD is reused for several different expansions, so the term is inherently polysemous rather than canonical (Kim et al., 2023, Wang et al., 2023, Yu et al., 2023, Decatur et al., 2023, Chachy et al., 12 Mar 2025, Lukoianov et al., 2024).

1. Terminology and scope

The first technical requirement is disambiguation. In the supplied literature, CSD does not denote a single formulation.

Usage in the literature	Meaning of CSD	Core control locus
"Collaborative Score Distillation" (Kim et al., 2023)	multi-sample SDS via SVGD	kernel-weighted score sharing and repulsion
"Canonical Score Distillation" (Wang et al., 2023)	canonical-field and warping-field distillation for 4D	explicit gradients through canonical generation and warping refinement
"Classifier Score Distillation" (Yu et al., 2023)	CFG-difference-only distillation	implicit classifier gradient
"Cascaded Score Distillation" (Decatur et al., 2023)	multi-stage diffusion supervision	stage-wise multi-resolution score aggregation
"Concrete Score Distillation" (Kim et al., 30 Sep 2025)	LLM distillation by concrete score matching	relative logit differences across vocabulary pairs

This terminological collision matters because each formulation changes a different part of the SDS pipeline. A plausible implication is that “control-side” is best reserved for the location of intervention—the score-distillation layer, the aggregation rule, or the structured conditioning path—rather than for any single acronym expansion.

2. SDS as the common substrate

Most of the visual formulations in this family inherit the standard SDS setup in which a differentiable generator or renderer $g(\theta)$ is optimized under a frozen text-conditioned diffusion prior. In that baseline, the image or rendering is $x_0 = g(\theta)$, the noisy sample is $x_t = \alpha_t x_0 + \sigma_t \epsilon_t$, and the SDS gradient is

$$\nabla_\theta L_{SDS}(x_0 = g(\theta)) = \mathbb{E}_{t,\epsilon_t} \left[ w(t) (\epsilon_\phi(x_t, y, t) - \epsilon_t) \frac{\partial x_0}{\partial \theta} \right].$$

RewardSDS states the key limitation of this baseline directly: standard SDS averages gradients over noise samples $\epsilon_t$ and timesteps $t$ uniformly, so every noise draw at a given timestep contributes equally even though some draws are more aligned than others (Chachy et al., 12 Mar 2025). The same paper also characterizes VSD as performing this procedure “in a variational/particle way using an auxiliary LoRA-tuned diffusion” (Chachy et al., 12 Mar 2025).

Under the control-side reading, the unifying move is not to replace the diffusion prior, but to change how that prior is used. This suggests a taxonomy in which the intervention can act on one of four axes: the interaction among samples, the factorization of gradients through structured latent controls, the composition of the score itself, or the Monte Carlo estimator over noises and timesteps.

3. Multi-sample and structured-control formulations

"Collaborative Score Distillation" generalizes SDS from one optimized sample to a set of jointly optimized samples, treating multiple images as particles in a Stein Variational Gradient Descent update. For particles $\{\theta_i\}_{i=1}^N$, rendered images $\mathbf{x}^{(i)} = g(\theta_i)$, and noisy images $\mathbf{x}_t^{(i)}$, its core gradient is

$\nabla_{\theta_i} \mathcal{L}_{\tt{CSD}\big(\theta_i\big) = \frac{w(t)}{N}\sum_{j=1}^N \left( k(\mathbf{x}_t^{(j)}, \mathbf{x}_t^{(i)}) \big(\boldsymbol{\epsilon}_\phi^{\omega}(\mathbf{x}_t^{(j)};y,t) - \boldsymbol{\epsilon} \big) + \nabla_{\mathbf{x}_t^{(j)}} k(\mathbf{x}_t^{(j)}, \mathbf{x}_t^{(i)}) \right)\frac{\partial \mathbf{x}^{(i)}}{\partial \theta_i}.$

The two decisive terms are the kernel-weighted averaging of per-particle scores and the repulsive gradient of the kernel. If $x_0 = g(\theta)$0, $x_0 = g(\theta)$1, and $x_0 = g(\theta)$2, the update reduces to SDS exactly. The mechanism is therefore an exact multi-particle generalization of SDS rather than a separate prior. Similar images exchange score information strongly, while the repulsive term prevents collapse (Kim et al., 2023).

"Canonical Score Distillation" changes a different part of the pipeline. AnimatableDreamer factorizes a 4D articulated model into a canonical model and a warping / motion field, then backpropagates diffusion supervision through both paths. Its core gradient is

$x_0 = g(\theta)$3

The paper explicitly names the two inner Jacobian factors canonical generation and warping refinement. In that formulation, CSD “lifts” 2D diffusion for “temporal consistent 4D generation,” while also refining bones and skinning so that articulated poses remain plausible under multi-view supervision (Wang et al., 2023).

Taken together, these two formulations show two distinct control-side strategies. Collaborative CSD imposes interaction between samples. Canonical CSD imposes structure inside the generator, using diffusion gradients to refine not only appearance-bearing variables but also bones, skinning weights, and warping fields.

4. Classifier-side guidance and cascaded supervision

"Classifier Score Distillation" reinterprets the practical success of SDS with classifier-free guidance. The key observation is that the CFG difference

$x_0 = g(\theta)$4

is proportional to the gradient of an implicit classifier for $x_0 = g(\theta)$5. CSD therefore discards the generative-prior term and retains only the classifier score, yielding

$x_0 = g(\theta)$6

The resulting view is explicitly classifier-side: diffusion is used as an implicit text-image classifier rather than as a conditional density to be matched by KL. Negative prompts become optimization against two implicit classifiers, and text-guided editing becomes a weighted difference between a target classifier score and an edit-attribute classifier score (Yu et al., 2023).

"Cascaded Score Distillation" alters the score-distillation side along a scale axis. 3D Paintbrush uses a cascaded text-to-image diffusion model $x_0 = g(\theta)$7, with stage 1 providing global semantic and positional supervision and later super-resolution stages providing high-frequency detail. For stage $x_0 = g(\theta)$8, the image-space gradient is

$x_0 = g(\theta)$9

and the full gradient combines stage-wise terms with user-set weights $x_t = \alpha_t x_0 + \sigma_t \epsilon_t$0. In 3D Paintbrush this supervision is applied jointly to a localization map, a local texture map, and a background texture map, so the diffusion prior supervises not only appearance but also where the edit should occur (Decatur et al., 2023).

These two formulations expose complementary control principles. Classifier CSD isolates the text-discriminative part of the score. Cascaded CSD structures supervision across multiple resolutions and multiple rendered branches.

5. Reward weighting and DDIM-consistent noise control

RewardSDS introduces control at the level of the Monte Carlo estimator over noises. For each rendered image $x_t = \alpha_t x_0 + \sigma_t \epsilon_t$1, the method samples multiple noises at one timestep, denoises each noisy image, scores the denoised images with a reward model $x_t = \alpha_t x_0 + \sigma_t \epsilon_t$2, converts the reward scores to weights, and forms a reward-weighted SDS gradient:

$x_t = \alpha_t x_0 + \sigma_t \epsilon_t$3

The important conceptual statement is explicit: control is applied on the sampling / weighting of gradient contributions on the “SDS side,” not on the denoising network architecture or parameters. The best-performing weighting scheme in the reported ablation assigns $x_t = \alpha_t x_0 + \sigma_t \epsilon_t$4 for the top 2 reward samples, $x_t = \alpha_t x_0 + \sigma_t \epsilon_t$5 for the bottom 2, and $x_t = \alpha_t x_0 + \sigma_t \epsilon_t$6 for all others (Chachy et al., 12 Mar 2025).

"Score Distillation via Reparametrized DDIM" moves control to the noise term itself. It argues that SDS is equivalent to a DDIM-like update in the $x_t = \alpha_t x_0 + \sigma_t \epsilon_t$7 space, but with freshly sampled i.i.d. noise at every step rather than a noise term consistent with the DDIM trajectory. The proposed SDI gradient is

$x_t = \alpha_t x_0 + \sigma_t \epsilon_t$8

where $x_t = \alpha_t x_0 + \sigma_t \epsilon_t$9 is approximated by conditional DDIM inversion plus a small Gaussian jitter. The paper’s central claim is that the excessive variance of SDS originates from replacing DDIM-consistent noise by i.i.d. noise, and that inversion-based noise recovery makes score distillation much closer to DDIM sampling (Lukoianov et al., 2024).

A plausible implication is that control-side score distillation includes not only changes to what score is distilled, but also changes to which stochastic realization is allowed to define that score.

6. Applications, empirical behavior, and unresolved terminology

The empirical record across these papers is broad. Collaborative CSD is reported on panorama image editing, video editing, 3D scene editing, and a DreamFusion-style text-to-3D ablation. In video editing, CSD-Edit reports higher CLIP directional (0.320 vs 0.314 / 0.230), higher CLIP image consistency (0.957 vs 0.948 / 0.949), and lower LPIPS (0.236 vs 0.267 / 0.283) than FateZero and Pix2Video. In 3D scene editing, it slightly improves all three reported metrics over IN2N: CLIP dir: 0.239 vs 0.230; CLIP img: 0.995 vs 0.994; LPIPS: 0.043 vs 0.048. In the DreamFusion-style ablation it improves FID (247.1 vs 259.4 for SDS baseline) and is described as reducing the Janus problem (Kim et al., 2023).

Canonical CSD is evaluated on both reconstruction and text-guided generation from monocular video. In monocular reconstruction, the ablation without $$\nabla_\theta L_{SDS}(x_0 = g(\theta)) = \mathbb{E}_{t,\epsilon_t} \left[ w(t) (\epsilon_\phi(x_t, y, t) - \epsilon_t) \frac{\partial x_0}{\partial \theta} \right].$$0 degrades Cat-Coco from CD 3.65 / F@2% 63.3 to CD 8.34 / F@2% 32.6. In text-to-4D generation, the full method reports CLIP: 38.2, CLIP-T: 96.6, R-Precision@10: 87.5, and GPT Eval3D Elo: 1098, with weaker results when $$\nabla_\theta L_{SDS}(x_0 = g(\theta)) = \mathbb{E}_{t,\epsilon_t} \left[ w(t) (\epsilon_\phi(x_t, y, t) - \epsilon_t) \frac{\partial x_0}{\partial \theta} \right].$$1, $$\nabla_\theta L_{SDS}(x_0 = g(\theta)) = \mathbb{E}_{t,\epsilon_t} \left[ w(t) (\epsilon_\phi(x_t, y, t) - \epsilon_t) \frac{\partial x_0}{\partial \theta} \right].$$2, or $$\nabla_\theta L_{SDS}(x_0 = g(\theta)) = \mathbb{E}_{t,\epsilon_t} \left[ w(t) (\epsilon_\phi(x_t, y, t) - \epsilon_t) \frac{\partial x_0}{\partial \theta} \right].$$3 are removed (Wang et al., 2023).

Classifier CSD reports a text-to-3D CLIP Score of 78.6, compared with 67.5 for DreamFusion and 74.9 for Magic3D, and a user preference of 59.4% over DreamFusion, Magic3D, and other baselines. In texture synthesis, user preference is 57.7% in favor of CSD, and the paper reports runtime of ~1 hour on a single A800 GPU for mesh refinement, versus ~8 hours for ProlificDreamer under similar conditions (Yu et al., 2023).

RewardSDS shows that reward weighting can be layered on top of existing SDS pipelines without changing the diffusion prior. With MVDream + NeRF, the reported changes are CLIP: 26.19 → 27.12, Aesthetic: 5.83 → 5.97, LLM-G: 5.86 → 6.07, Alignment: 3.51 → 4.21, and Realism: 3.14 → 3.79. With MVDream + 3D Gaussians, the reported changes are CLIP: 24.71 → 25.24, Aesthetic: 5.73 → 5.79, LLM-G: 4.90 → 5.31, User MOS alignment: 2.74 → 4.11, and Realism: 2.28 → 3.13 (Chachy et al., 12 Mar 2025).

Reparametrized DDIM provides a different empirical profile: it does not add multi-view interaction or reward models, but controls the noise process itself. The paper reports CLIP Score: SDI ~33.47 vs SDS ~29.81 and Divergence rate: SDI ~4.7% vs SDS ~18.6%, alongside better CLIP IQA and lower instability (Lukoianov et al., 2024).

These gains come with recurring limitations. Collaborative CSD inherits all pairwise kernel evaluations in the batch, i.e. $$\nabla_\theta L_{SDS}(x_0 = g(\theta)) = \mathbb{E}_{t,\epsilon_t} \left[ w(t) (\epsilon_\phi(x_t, y, t) - \epsilon_t) \frac{\partial x_0}{\partial \theta} \right].$$4, and LPIPS-based kernels are more expensive than $$\nabla_\theta L_{SDS}(x_0 = g(\theta)) = \mathbb{E}_{t,\epsilon_t} \left[ w(t) (\epsilon_\phi(x_t, y, t) - \epsilon_t) \frac{\partial x_0}{\partial \theta} \right].$$5-based kernels (Kim et al., 2023). Canonical CSD requires rendering multiple views and backpropagating through camera, warping, and canonical field, which the paper identifies as high memory and compute (Wang et al., 2023). Classifier CSD explicitly notes that direct application to pure 2D image optimization produces artifacts, even though the same mechanism works well in multi-view 3D (Yu et al., 2023). RewardSDS adds repeated denoising and reward-model evaluation, with runtime increasing from ~45s per image for baseline SDS to ~2227s per image for a large-scale configuration (Chachy et al., 12 Mar 2025). SDI requires inversion steps and careful guidance schedules, which increases VRAM and compute relative to SDS even though it remains lighter than heavy multi-stage baselines (Lukoianov et al., 2024).

One final ambiguity should be kept explicit. In a separate line of work outside visual diffusion, "Concrete Score Distillation" uses the same acronym CSD for a discrete score-matching objective in autoregressive LLM distillation. There, CSD aligns relative logit differences across all vocabulary pairs between student and teacher with flexible weighting, and the object of control is not a diffusion-guided renderer but a token-level KD objective (Kim et al., 30 Sep 2025). This separate usage reinforces the broader point: CSD is an overloaded acronym, whereas control-side score distillation is best treated as a description of where the intervention occurs in a score-distillation pipeline rather than as a single, settled method name.