Video Score Distillation Sampling Explained

• Video Score Distillation Sampling extends image SDS to video sequences, enabling joint optimization of frame coherence and temporal dynamics.
• It leverages paired denoising score objectives and adaptive smoothing techniques to correct noise and preserve source structure in video editing tasks.
• Applications include text-to-video editing, 4D generation, and student-model distillation for rapid synthesis with improved consistency and efficiency.

Video Score Distillation Sampling denotes a family of optimization and distillation procedures that extend score distillation from single images to temporally ordered visual data. In its basic form, a video or stacked set of rendered frames is noised at a diffusion timestep and scored by a pretrained text-conditioned diffusion model; the discrepancy between predicted and reference noise, or between paired denoisers, is then backpropagated into editable video frames, latent codes, or 3D/4D scene parameters. Across recent work, the term encompasses direct video editing, collaborative multi-frame optimization, hybrid image-video supervision for text-to-4D generation, motion distillation for static 3D assets, and student-model distillation for few-step video synthesis (Jeong et al., 2024, Zhu et al., 2024, Kim et al., 2023, Bahmani et al., 2023, Sun et al., 14 Dec 2025, Ding et al., 2024, Zheng et al., 9 Oct 2025).

1. Foundational formulation

The canonical starting point is the SDS objective. Given a clean render or frame sequence $x_0$, one samples a timestep $t$ and Gaussian noise $\epsilon$, forms

$$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,$$

and queries a pretrained denoiser conditioned on text. In the single-image setting, the resulting gradient has the form

$$\nabla_\phi \mathcal L_{\rm SDS} = \mathbb E_{t,\epsilon} \Big[ w(t)\big(\epsilon_\theta(x_t,t,c)-\epsilon\big)\, \frac{\partial x_t}{\partial \phi} \Big],$$

where $\phi$ parameterizes the object being optimized. This same template appears in video-oriented formulations, but the optimized variable may be a stack of frames, a latent video tensor, or a differentiable 3D or 4D representation (Sun et al., 14 Dec 2025, Bahmani et al., 2023).

For video, one straightforward extension is to render or encode $N$ frames jointly, stack them into $X=[x^{(1)},\ldots,x^{(V)}]^\top$, add noise to all frames, and backpropagate a video-model discrepancy over the entire sequence:

$$\nabla_\phi L_{\rm VID} = \mathbb E_{t,\mathcal E} \Big[ w(t)\big(\epsilon_\theta(Z_t;t,y)-\mathcal E\big)\, \frac{\partial X}{\partial \phi} \Big].$$

This preserves the SDS logic while making the diffusion prior explicitly spatiotemporal rather than framewise (Bahmani et al., 2023).

Editing-oriented variants often replace pure-noise matching with a paired-branch formulation. DreamMotion uses Delta Denoising Score and defines a video loss

$$L_{\rm V\text{-}DDS}(\theta;y) = \mathbb E_{t,\epsilon} \Bigl\| \epsilon_\phi^w\bigl(x_t^{1:N}(\theta),t,y\bigr) - \epsilon_\phi^w\bigl(\hat x_t^{1:N},t,\hat y\bigr) \Bigr\|_2^2,$$

where $t$0 is the source video and $t$1 is the target caption. Zero-shot video editing through Adaptive Sliding Score Distillation likewise introduces an editing branch and a reference branch over a shared timestep and noise realization (Jeong et al., 2024, Zhu et al., 2024).

A recurrent distinction is therefore between noise-reconstruction objectives, which compare denoiser output to sampled Gaussian noise, and paired-score objectives, which compare denoiser outputs from source and target conditions. The latter are used to reduce oversaturation, blurriness, or source-structure drift in video editing (Jeong et al., 2024, Zhu et al., 2024).

2. Spatiotemporal failure modes and why naive video SDS is difficult

The principal technical difficulty is that temporal coherence is not automatically induced by applying image-style SDS to each frame. DreamMotion reports that video score distillation can effectively introduce new content indicated by target text, but can also cause significant structure and motion deviation. Zero-Shot Video Editing through Adaptive Sliding Score Distillation states that prompt-guided denoising inherently compromises the original video's integrity, resulting in unintended over-editing and temporal discontinuities, and further notes that unstable noise predictions from current open-source T2V models yield “contaminations” in edited regions (Jeong et al., 2024, Zhu et al., 2024).

Related work identifies additional failure modes in other regimes. In Animus3D, previous methods that leverage vanilla SDS to distill motion from pretrained text-to-video diffusion lead to animations with minimal movement or noticeable jitter; off-the-shelf video diffusion models prompted with “a static statue, no motion” still hallucinate small motions. In large-scale consistency distillation, pure continuous-time consistency is reported to have fundamental quality limitations in fine-detail generation, with blurred or distorted details attributed to error accumulation and the “mode-covering” nature of the forward-divergence objective (Sun et al., 14 Dec 2025, Zheng et al., 9 Oct 2025).

Text-to-4D generation introduces another coupled difficulty: existing methods face a three-way tradeoff between scene appearance, 3D structure, and motion. Image models provide realistic appearance and 3D structure but no motion, while text-to-video models provide motion but poorer appearance and 3D structure (Bahmani et al., 2023).

These observations collectively suggest that “video SDS” is not a single algorithmic pattern but a problem class defined by how score information is regularized, paired, or redistributed across time. A plausible implication is that successful methods must specify not only the score term itself but also the temporal coupling mechanism by which that score is made compatible with motion, geometry, and appearance constraints.

3. Consistency-preserving variants for video editing

Several prominent methods modify the basic objective specifically to preserve source-video structure and motion during editing. DreamMotion augments video-DDS with space-time self-similarity matching in U-Net features. Spatial self-similarity is computed within each frame from decoder-stage key tensors, while temporal self-similarity is computed from framewise spatial summaries. The total loss is

$t$2

with typical weights $t$3. All three losses share the same draw of $t$4 and $t$5, so only one forward U-Net pass per iteration is needed. On DAVIS/WebVid with 8–16 frame clips and two pipelines, DreamMotion reports non-cascaded Text-Align $t$6 versus a best-baseline $t$7, Frame-Cons $t$8 versus $t$9, and Human Structure & Motion Preservation $\epsilon$0 versus $\epsilon$1; in the cascaded Show-1 setting, Structure & Motion Preservation is $\epsilon$2 versus a next-best $\epsilon$3 (Jeong et al., 2024).

Adaptive Sliding Score Distillation addresses noisy reference-branch predictions by temporally smoothing them in a 3-frame sliding window and applying that smoothing only in masked regions where editing/reference differences are large. The smoothed reference branch is

$\epsilon$4

with $\epsilon$5 as the illustrative setting, and the final per-frame gradient blends smoothed and unsmoothed reference predictions under an adaptive mask. The method further adds Weighted Attention Fusion, which convexly blends self-attention from the editing branch with reference attention from the source branch, and supplements T2V guidance with a per-frame Stable Diffusion DDS gradient. Against naive Video-DDS, the reported auto-metrics improve from CLIP-Text Alignment $\epsilon$6 to $\epsilon$7 and CLIP-Frame Consistency $\epsilon$8 to $\epsilon$9; user-study “ranked best” rates rise to $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,$$0 for prompt alignment, $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,$$1 for temporal consistency, and $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,$$2 for preservation of unedited content (Zhu et al., 2024).

Collaborative Score Distillation takes a different route and treats frames as Stein Variational Gradient Descent particles. Each frame receives an update that mixes an attraction term, based on all other frame scores, with a repulsion term that prevents collapse:

$$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,$$3

In video editing on DAVIS 2017, CSD-Edit reports CLIP Directional Similarity $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,$$4, CLIP Image Consistency $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,$$5, and LPIPS between adjacent frames $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,$$6, outperforming FateZero and Pix2Vid on the same table. The paper attributes consistent edits to the fact that every frame’s update depends on all others (Kim et al., 2023).

Method	Consistency mechanism	Reported effect
DreamMotion	S-SSM + T-SSM in U-Net features	Higher structure/motion preservation
ASSD	Sliding-window smoothing + masking + attention fusion	Fewer contaminations, better temporal consistency
CSD	SVGD attraction/repulsion across frames	More stable multi-frame edits

4. Motion-aware score distillation in 3D and 4D

In 4D generation, video score distillation is embedded in a broader multi-prior optimization problem. 4D-fy renders multiple frames from a single 4D representation, applies video SDS to all rendered frames jointly, and alternates that signal with 3D-aware image SDS and variational SDS from a finetuned image model. The hybrid objective is described as an alternating schedule whose expected effect is equivalent to

$$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,$$7

with $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,$$8. In practice, Stage 3 of the algorithm reintroduces video updates after a static warm-up and appearance finetune, uses video timesteps sampled from $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,$$9, and renders 16 frames at $$\nabla_\phi \mathcal L_{\rm SDS} = \mathbb E_{t,\epsilon} \Big[ w(t)\big(\epsilon_\theta(x_t,t,c)-\epsilon\big)\, \frac{\partial x_t}{\partial \phi} \Big],$$0 before upsampling to $$\nabla_\phi \mathcal L_{\rm SDS} = \mathbb E_{t,\epsilon} \Big[ w(t)\big(\epsilon_\theta(x_t,t,c)-\epsilon\big)\, \frac{\partial x_t}{\partial \phi} \Big],$$1 for Zeroscope. The stated motivation is to resolve the tradeoff among geometry, appearance quality, and motion realism (Bahmani et al., 2023).

Animus3D specializes the video-distillation idea to text-driven 3D animation from a static asset. Its core claim is that motion distillation should transport from a static video distribution to a dynamic one rather than from pure noise. It therefore defines a motion score

$$\nabla_\phi \mathcal L_{\rm SDS} = \mathbb E_{t,\epsilon} \Big[ w(t)\big(\epsilon_\theta(x_t,t,c)-\epsilon\big)\, \frac{\partial x_t}{\partial \phi} \Big],$$2

and uses the gradient

$$\nabla_\phi \mathcal L_{\rm SDS} = \mathbb E_{t,\epsilon} \Big[ w(t)\big(\epsilon_\theta(x_t,t,c)-\epsilon\big)\, \frac{\partial x_t}{\partial \phi} \Big],$$3

The static branch is created by LoRA fine-tuning of the video denoiser on short static video clips with a “no-motion” text prompt. To preserve appearance, the method replaces random-noise perturbation with DDIM inversion so that both branches are computed on a deterministic noising path. It further regularizes 3D Gaussian splat trajectories with a temporal smoothness loss $$\nabla_\phi \mathcal L_{\rm SDS} = \mathbb E_{t,\epsilon} \Big[ w(t)\big(\epsilon_\theta(x_t,t,c)-\epsilon\big)\, \frac{\partial x_t}{\partial \phi} \Big],$$4 and a spatial rigidity As-Rigid-As-Possible loss $$\nabla_\phi \mathcal L_{\rm SDS} = \mathbb E_{t,\epsilon} \Big[ w(t)\big(\epsilon_\theta(x_t,t,c)-\epsilon\big)\, \frac{\partial x_t}{\partial \phi} \Big],$$5, then adds a motion refinement module that temporally interpolates from $$\nabla_\phi \mathcal L_{\rm SDS} = \mathbb E_{t,\epsilon} \Big[ w(t)\big(\epsilon_\theta(x_t,t,c)-\epsilon\big)\, \frac{\partial x_t}{\partial \phi} \Big],$$6 to $$\nabla_\phi \mathcal L_{\rm SDS} = \mathbb E_{t,\epsilon} \Big[ w(t)\big(\epsilon_\theta(x_t,t,c)-\epsilon\big)\, \frac{\partial x_t}{\partial \phi} \Big],$$7 frames and applies rectified-flow or diffusion-based SDEdit before pulling the 3D motion field toward the refined sequence with an $$\nabla_\phi \mathcal L_{\rm SDS} = \mathbb E_{t,\epsilon} \Big[ w(t)\big(\epsilon_\theta(x_t,t,c)-\epsilon\big)\, \frac{\partial x_t}{\partial \phi} \Big],$$8 loss. The reported outcome is more substantial and detailed motion than state-of-the-art baselines while maintaining high visual integrity (Sun et al., 14 Dec 2025).

Within this 3D/4D setting, video score distillation ceases to be only an editing loss on pixels or latents; it becomes a supervision mechanism for dynamic scene representations. This suggests that the quality of the “video prior” depends as much on the choice of source distribution and geometric regularization as on the score term itself.

5. From optimization-time supervision to student-model distillation

A second line of work uses score distillation not to optimize a single video instance, but to train a fast student generator. DOLLAR distills a pretrained teacher video diffusion model into a student that samples in $$\nabla_\phi \mathcal L_{\rm SDS} = \mathbb E_{t,\epsilon} \Big[ w(t)\big(\epsilon_\theta(x_t,t,c)-\epsilon\big)\, \frac{\partial x_t}{\partial \phi} \Big],$$9–$\phi$0 denoising steps by combining Variational Score Distillation, Consistency Distillation, and Latent Reward Model fine-tuning. The overall objective is

$\phi$1

with $\phi$2 and $\phi$3. The student is evaluated on 10-second videos with 128 frames at 12 FPS. Reported VBench Total Scores are $\phi$4 for the 50-step DDIM teacher, $\phi$5 for DOLLAR with PickScore LRM in 4 steps, and $\phi$6 for DOLLAR with HPSv2 LRM in 4 steps. Reported diffusion-time speedups are approximately $\phi$7 for 4 steps, $\phi$8 for 2 steps, and $\phi$9 for 1 step; total-inference speedups are approximately $N$0, $N$1, and $N$2, respectively. The paper also reports that VSD only yields a Vendi diversity score of $N$3, while VSD+CD gives $N$4, relative to a teacher score of $N$5 (Ding et al., 2024).

Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency extends this acceleration agenda to application-scale image and video models. It frames the student as a continuous-time consistency function $N$6 trained with an sCM objective, then adds a DMD-style score-distillation regularizer:

$N$7

with empirical $N$8. The stated rationale is that sCM alone is forward-KL, mode-covering, and prone to blurred or distorted fine details, while the added reverse-divergence term sharpens details without sacrificing diversity. The method introduces a FlashAttention-2 JVP kernel to make consistency training compatible with large models and video-scale Jacobian-vector products. On VBench at 480p, the 1.3B rCM at 4 steps reports a Total Score of $N$9 versus a teacher score of $X=[x^{(1)},\ldots,x^{(V)}]^\top$0, with throughput $X=[x^{(1)},\ldots,x^{(V)}]^\top$1 FPS versus $X=[x^{(1)},\ldots,x^{(V)}]^\top$2 FPS, or approximately $X=[x^{(1)},\ldots,x^{(V)}]^\top$3 speedup. On 14B, rCM at 4 steps reports $X=[x^{(1)},\ldots,x^{(V)}]^\top$4 versus a teacher score of $X=[x^{(1)},\ldots,x^{(V)}]^\top$5, with throughput $X=[x^{(1)},\ldots,x^{(V)}]^\top$6 FPS versus $X=[x^{(1)},\ldots,x^{(V)}]^\top$7 FPS, or approximately $X=[x^{(1)},\ldots,x^{(V)}]^\top$8; at 2 steps it reaches $X=[x^{(1)},\ldots,x^{(V)}]^\top$9 FPS, approximately $$\nabla_\phi L_{\rm VID} = \mathbb E_{t,\mathcal E} \Big[ w(t)\big(\epsilon_\theta(Z_t;t,y)-\mathcal E\big)\, \frac{\partial X}{\partial \phi} \Big].$$0, and at 1 step $$\nabla_\phi L_{\rm VID} = \mathbb E_{t,\mathcal E} \Big[ w(t)\big(\epsilon_\theta(Z_t;t,y)-\mathcal E\big)\, \frac{\partial X}{\partial \phi} \Big].$$1 FPS, approximately $$\nabla_\phi L_{\rm VID} = \mathbb E_{t,\mathcal E} \Big[ w(t)\big(\epsilon_\theta(Z_t;t,y)-\mathcal E\big)\, \frac{\partial X}{\partial \phi} \Big].$$2 (Zheng et al., 9 Oct 2025).

These methods shift the role of score distillation from test-time optimization to training-time transfer. A plausible implication is that video score distillation now spans both instance-specific editing frameworks and general-purpose generative model compression.

6. Conceptual boundaries, misconceptions, and unresolved tradeoffs

A common misconception is that video score distillation is simply image SDS applied independently to each frame. The literature instead shows multiple distinct coupling mechanisms: feature-space space-time self-similarity, adaptive sliding-window smoothing, attention fusion with a reference branch, SVGD attraction-repulsion among frame particles, alternating image/video priors in 4D optimization, and explicit static-versus-dynamic score subtraction in motion distillation (Jeong et al., 2024, Zhu et al., 2024, Kim et al., 2023, Bahmani et al., 2023, Sun et al., 14 Dec 2025).

Another misconception is that score distillation inherently preserves appearance or source motion. Standard SDS uses a random draw of $$\nabla_\phi L_{\rm VID} = \mathbb E_{t,\mathcal E} \Big[ w(t)\big(\epsilon_\theta(Z_t;t,y)-\mathcal E\big)\, \frac{\partial X}{\partial \phi} \Big].$$3, and Animus3D explicitly states that this random draw can disturb appearance, motivating DDIM inversion for deterministic noising. DreamMotion and ASSD both add preservation terms because target-conditioned score distillation alone can deviate from original structure and motion; ASSD further restricts smoothing to masked high-change regions to avoid over-blurring (Sun et al., 14 Dec 2025, Jeong et al., 2024, Zhu et al., 2024).

The field also exhibits persistent tradeoffs rather than a single dominant resolution. 4D-fy identifies a three-way tradeoff between appearance, 3D structure, and motion. DOLLAR reports that VSD only can suffer from mode collapse in diversity metrics. rCM attributes fine-detail degradation in pure consistency training to forward-divergence behavior, and corrects it with a reverse-divergence score regularizer. ASSD identifies unstable T2V noise predictions as a cause of editing artifacts, while Animus3D identifies the inability of nominally static priors to remain truly static as a source of motion hallucination (Bahmani et al., 2023, Ding et al., 2024, Zheng et al., 9 Oct 2025, Zhu et al., 2024, Sun et al., 14 Dec 2025).

Taken together, these results define Video Score Distillation Sampling less as a fixed recipe than as a design space for reconciling diffusion-model priors with temporal structure, source preservation, dynamic geometry, and computational efficiency. This suggests that future progress is likely to come from better score pairing, better temporal coupling, and more precise control over the divergence being optimized, rather than from scaling a vanilla SDS objective alone.