DragStream: Training-Free Interactive Video Manipulation

• DragStream is a training-free add-on that introduces real-time, drag-oriented control into autoregressive video diffusion models by enabling interactive editing and animation.
• It mitigates latent distribution drift and context interference using Adaptive Distribution Self-Rectification (ADSR) and Spatial-Frequency Selective Optimization (SFSO), ensuring stable latent manipulations.
• Experimental validation on the REVEL task shows that DragStream outperforms prior methods with lower ObjMC, FID, FVD, and DAI scores, delivering smoother motion and fewer artifacts.

DragStream is a training-free add-on to autoregressive Video Diffusion Models (VDMs) for streaming, fine-grained, drag-oriented interactive video manipulation during generation. It is introduced as the method for resolving the REVEL task—stReaming drag-oriEnted interactiVe vidEo manipuLation—which enables users to modify generated videos anytime on anything via fine-grained, interactive drag, with support for editing and animating video frames under user-specified translation, deformation, and rotation effects (Zhou et al., 3 Oct 2025). Within that formulation, DragStream addresses two failure modes observed during latent-region optimization in autoregressive generation: latent distribution drift induced by repeated drag perturbations, and context interference from cached context frames that produces visually unnatural outcomes. Its two core components are Adaptive Distribution Self-Rectification (ADSR) and Spatial-Frequency Selective Optimization (SFSO), both integrated directly into the inference loop without updating model weights (Zhou et al., 3 Oct 2025).

1. REVEL task and problem setting

REVEL extends the standard autoregressive VDM pipeline by injecting interactive drag control into the generation process. At each generation step $k$, an autoregressive VDM produces a new frame $\Gamma^k$ conditioned on the previous $L_c$ frames. A user may then click–drag any region $H_i^k$ in $\Gamma^k$ and specify an operation, encoded as $U^k=(E^k,C^k)$. The operation may correspond to translation, deformation, or rotation. If $\eta^k=\mathrm{Editing}$, the current frame is re-denoised; if $\eta^k=\mathrm{Animation}$, one or more future frames $\Gamma^{k'}$, $k'>k$, are synthesized along the trajectory (Zhou et al., 3 Oct 2025).

The paper characterizes REVEL as a unification of drag-style video manipulation beyond DragVideo and SG-I2V, explicitly combining editing and animation under a single framework and supporting both user-specified translation, deformation, and rotation effects (Zhou et al., 3 Oct 2025). A plausible implication is that REVEL is intended as a task-level abstraction rather than a narrowly scoped interface for one manipulation mode.

The central difficulty is not merely adding a drag handle to an existing diffusion interface. The paper identifies two specific obstacles in streaming autoregressive settings. First, drag-induced perturbations accumulate in latent space, causing severe latent distribution drift that can halt the drag process. Second, streaming drag is easily disturbed by context frames, which leads to unnatural outcomes (Zhou et al., 3 Oct 2025). These two observations motivate the design of DragStream.

2. Latent-region optimization and failure modes

DragStream intercepts the latent code $\Gamma^k$0 at a late denoising step $\Gamma^k$1, with the implementation example $\Gamma^k$2 of 4, and extracts multi-layer features $\Gamma^k$3 from the transformer denoiser $\Gamma^k$4. The target region is constructed through

$\Gamma^k$5

which 2D-rotates or translates $\Gamma^k$6 into a target mask $\Gamma^k$7 (Zhou et al., 3 Oct 2025).

The latent is then optimized using a reconstruction term on the edited region and a consistency term on the unedited region $\Gamma^k$8:

$\Gamma^k$9

and

$L_c$0

These losses drive updates through $L_c$1 (Zhou et al., 3 Oct 2025).

Without additional regularization, the paper reports two characteristic pathologies. The first, Latent Distribution Drift (Challenge 1), arises because repeated region-wise pushes shift the marginal distribution of $L_c$2 away from $L_c$3, leading to color shifts or collapsed drags. The second, Context Interference (Challenge 2), occurs when high-frequency details from cached context frames bleed into the current optimization, producing duplicated artifacts or unnatural shapes (Zhou et al., 3 Oct 2025).

These observations define the technical scope of DragStream. It does not replace the underlying autoregressive VDM; instead, it wraps ADSR and SFSO around each optimization iteration and then resumes denoising and KV-cache updating in the usual autoregressive loop (Zhou et al., 3 Oct 2025).

3. Adaptive Distribution Self-Rectification

Adaptive Distribution Self-Rectification is designed to counter latent distribution drift. After each latent update,

$L_c$4

ADSR re-normalizes the latent mean $L_c$5 and standard deviation $L_c$6 to match the running statistics of the $L_c$7 immediate predecessor frames (Zhou et al., 3 Oct 2025).

The running statistics are defined as

$L_c$8

The rectified latent is then

$L_c$9

By “snapping” each iteration back to a stable latent distribution, ADSR is reported to prevent drifted colors, collapsed shapes, or halts in the drag process (Zhou et al., 3 Oct 2025). In the implementation defaults, $H_i^k$0 neighboring frames are used (Zhou et al., 3 Oct 2025).

Conceptually, ADSR is a local statistical constraint rather than a learned correction model. Because it uses neighboring frames’ statistics, it preserves the training-free property of the method. This suggests that the method leverages temporal locality in the autoregressive stream as a proxy for latent-distribution regularity, rather than introducing a separate alignment objective.

4. Spatial-Frequency Selective Optimization

SFSO addresses context interference while still exploiting contextual information from cached frames. It operates in two stages during each latent-region optimization step: Switchable Frequency-domain Selection (SFS) and Criticality-driven Spatial-domain Selection (CSS) (Zhou et al., 3 Oct 2025).

Switchable Frequency-domain Selection

In each self-attention layer $H_i^k$1 used for feature extraction, with the example $H_i^k$2, the cached keys and values from context frames $H_i^k$3 are concatenated with the current $H_i^k$4. A random-band Butterworth filter is then applied in the Fourier domain before the inverse transform:

1. $H_i^k$5
2. $H_i^k$6, and similarly for $H_i^k$7
3. $H_i^k$8, with $H_i^k$9 chosen uniformly from a preset $\Gamma^k$0
4. $\Gamma^k$1

By randomly switching $\Gamma^k$2 each iteration, the method ensures a mixture of low- and mid-frequency context information propagates, mitigating both blur when $\Gamma^k$3 is too small and ringing or noise when $\Gamma^k$4 (Zhou et al., 3 Oct 2025).

Criticality-driven Spatial-domain Selection

Even after frequency-domain filtering, the paper notes that gradients can leak to background pixels. CSS therefore weights the gradient update by a Gaussian map $\Gamma^k$5 centered on the drag handle:

$\Gamma^k$6

with

$\Gamma^k$7

The update becomes

$\Gamma^k$8

The stated function of CSS is to confine high-magnitude gradient corrections to the immediate vicinity of the handle, thereby preserving background fidelity (Zhou et al., 3 Oct 2025).

Taken together, SFS and CSS form a selective propagation mechanism across frequency and space. The paper’s terminology emphasizes selection rather than simple suppression: contextual information is not removed wholesale, but filtered and spatially localized.

5. Integration into autoregressive video diffusion models

DragStream is described as being seamlessly integrated into existing autoregressive VDMs and implemented entirely inside the inference loop (Zhou et al., 3 Oct 2025). The paper reports integration with Self-Forcing, which uses 4 denoising steps, and notes appendix experiments on CausVid without finetuning (Zhou et al., 3 Oct 2025).

The operational pipeline is explicit. During autoregressive inference, at the chosen $\Gamma^k$9, one forks $U^k=(E^k,C^k)$0 into the latent optimizer, runs $U^k=(E^k,C^k)$1 gradient iterations per trajectory point with $U^k=(E^k,C^k)$2, using learning rate $U^k=(E^k,C^k)$3 and AdamW, applies ADSR and SFSO at each iteration, and then proceeds to downstream denoising steps and caches $U^k=(E^k,C^k)$4 for future frames (Zhou et al., 3 Oct 2025). The default hyperparameters reported are $U^k=(E^k,C^k)$5, $U^k=(E^k,C^k)$6, $U^k=(E^k,C^k)$7, and $U^k=(E^k,C^k)$8, and the method is said to be robust to small variations around these defaults (Zhou et al., 3 Oct 2025).

A central architectural property is that no weights of the VDM are updated. The paper states that all computation sits in the inference loop, and that the plug-and-play design preserves both the original generative model’s quality and its autoregressive speed because KV caching remains intact (Zhou et al., 3 Oct 2025). This suggests that DragStream is intended as an inference-time control layer rather than a retrained editing backbone.

6. Experimental validation, ablations, and scope

The experimental benchmark consists of 204 Self-Forcing–generated clips annotated with diverse drag trajectories and used as the REVEL benchmark (Zhou et al., 3 Oct 2025). Evaluation uses four metrics. ObjMC measures the average deviation between the user-specified trajectory and the object’s tracked path, with lower values better. DAI quantifies the $U^k=(E^k,C^k)$9 difference in latent patches around handle points, also with lower values better. FID and FVD are used to gauge overall per-frame and temporal realism, respectively, again with lower values indicating better performance (Zhou et al., 3 Oct 2025).

The paper compares DragStream to two adapted prior training-free drag methods, SG-I2V and DragVideo, under streaming inference. It states that DragStream wins decisively on all four metrics. The examples given are ObjMC dropping from approximately $\eta^k=\mathrm{Editing}$0 for SG-I2V/DragVideo to approximately $\eta^k=\mathrm{Editing}$1, FVD from approximately $\eta^k=\mathrm{Editing}$2 to approximately $\eta^k=\mathrm{Editing}$3, FID from approximately $\eta^k=\mathrm{Editing}$4 to approximately $\eta^k=\mathrm{Editing}$5, and DAI from approximately $\eta^k=\mathrm{Editing}$6 to approximately $\eta^k=\mathrm{Editing}$7 (Zhou et al., 3 Oct 2025). The accompanying visual description is “fewer artifacts, faithful object appearance, and smooth motion” (Zhou et al., 3 Oct 2025).

The ablation studies are structurally important because they delimit the roles of the components. Removing one of ADSR or SFSO degrades ObjMC, FVD, FID, and DAI; removing both collapses dragging entirely. Within SFSO, SFS alone or CSS alone is inferior to their conjunction (Zhou et al., 3 Oct 2025). The reported frequency-cutoff study further states that fixing $\eta^k=\mathrm{Editing}$8 yields blur and loss of detail, while $\eta^k=\mathrm{Editing}$9 yields ringing artifacts; random switching among $\eta^k=\mathrm{Animation}$0 gives the best trade-off (Zhou et al., 3 Oct 2025).

The optimization-loss study is similarly explicit. Dropping $\eta^k=\mathrm{Animation}$1 prevents region alignment, with ObjMC going to 90, while dropping $\eta^k=\mathrm{Animation}$2 reintroduces background artifacts, with FID going to 33; both terms are described as essential (Zhou et al., 3 Oct 2025). These observations constrain a common misconception that drag manipulation quality is determined mainly by the target-region objective: in the reported experiments, preservation of the unedited region is equally necessary for stable visual quality.

7. Position within interactive video manipulation

Within the paper’s framing, DragStream occupies the intersection of autoregressive video diffusion, drag-based interaction, and training-free inference-time control (Zhou et al., 3 Oct 2025). It is explicitly positioned against the challenge of achieving streaming, fine-grained control over autoregressive video diffusion outputs, and against the limitations of prior drag methods when adapted to streaming inference (Zhou et al., 3 Oct 2025).

Its defining characteristics are therefore not only functional but methodological. First, it targets streaming generation rather than offline post hoc editing. Second, it unifies editing and animation under the REVEL task. Third, it preserves a training-free, plug-and-play integration strategy. Fourth, its dual mechanism—statistical self-rectification in latent space and spatial-frequency-aware gradient projection in attention-guided optimization—addresses the two diagnosed failure modes directly (Zhou et al., 3 Oct 2025).

The paper concludes that DragStream equips any autoregressive VDM with fine-grained, on-the-fly drag control at no training cost, and that its combination of statistical self-rectification and spatial-frequency-aware gradient projection stabilizes latent manipulations, suppresses context interference, and achieves state-of-the-art editing and animation under REVEL (Zhou et al., 3 Oct 2025). A plausible implication is that the method’s broader significance lies in reframing interactive controllability for autoregressive video generators as an inference-time systems problem, rather than solely a model-training problem.