Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising (2511.08633v1)

Abstract: Diffusion-based video generation can create realistic videos, yet existing image- and text-based conditioning fails to offer precise motion control. Prior methods for motion-conditioned synthesis typically require model-specific fine-tuning, which is computationally expensive and restrictive. We introduce Time-to-Move (TTM), a training-free, plug-and-play framework for motion- and appearance-controlled video generation with image-to-video (I2V) diffusion models. Our key insight is to use crude reference animations obtained through user-friendly manipulations such as cut-and-drag or depth-based reprojection. Motivated by SDEdit's use of coarse layout cues for image editing, we treat the crude animations as coarse motion cues and adapt the mechanism to the video domain. We preserve appearance with image conditioning and introduce dual-clock denoising, a region-dependent strategy that enforces strong alignment in motion-specified regions while allowing flexibility elsewhere, balancing fidelity to user intent with natural dynamics. This lightweight modification of the sampling process incurs no additional training or runtime cost and is compatible with any backbone. Extensive experiments on object and camera motion benchmarks show that TTM matches or exceeds existing training-based baselines in realism and motion control. Beyond this, TTM introduces a unique capability: precise appearance control through pixel-level conditioning, exceeding the limits of text-only prompting. Visit our project page for video examples and code: https://time-to-move.github.io/.

• The paper introduces a training-free, plug-and-play framework for precise motion control in image-to-video diffusion via dual-clock denoising.
• It leverages coarse user-provided animations as motion proxies to enforce region-specific control without compromising visual fidelity.
• Experimental evaluations show reduced tracking errors and significant metric improvements (e.g., lower MSE and FID) over existing methods.

Time-to-Move: Training-Free, Precise, and Plug-and-Play Motion Control in Video Diffusion

Introduction

Time-to-Move (TTM) introduces a training-free, architecture-agnostic framework for precise, interactive motion control in image-to-video (I2V) diffusion models. Unlike prior approaches that require training or model modification, TTM leverages simple user-provided animations—such as cut-and-drag manipulations or depth-based reprojections—as coarse but explicit motion references. Critically, TTM’s dual-clock denoising process enforces strong adherence in user-specified regions while permitting natural adaptation elsewhere, enabling fine-grained control of both object and camera motion without degrading visual fidelity or incurring additional computational cost.

Figure 2: Overview of the TTM pipeline. Pixels inside the mask (motion-controlled regions) are denoised starting from lower noise, enforcing adherence to prescribed motion; outside regions start from higher noise, allowing for more natural scene adaptation.

Methodology

Crude Animation as Motion Proxy

The framework accepts a single reference image $I$, a coarsely warped video $V^w$ generated via user manipulation (e.g., cut-and-drag, forward warping using estimated monocular depth), and a binary mask $M$ indicating regions under explicit motion control. The warped video need only convey the trajectory and initial placement of the object; visual fidelity is immaterial. Any holes or disocclusions after warping are inpainted using simple heuristics. This approach supports diverse interactions, including translation, rotation, scaling, or viewpoint changes.

SDEdit Adaptation for Video Diffusion

Drawing inspiration from SDEdit, the TTM pipeline injects the motion signal by corrupting $V^w$ to an intermediate noise level $t^*$ and initializing sampling from this noisy trajectory:

$x_{t^*} \sim q(x_{t^*}\mid V^w)$

Sampling then proceeds conditioned additionally on $I$ using standard image-conditioned I2V diffusion backbones:

$x_0 \sim p_\theta(x_0 \mid x_{t^*}, I)$

The single-image condition preserves identity and appearance, while the noisy $V^w$ provides explicit motion.

Dual-Clock Denoising: Region-Conditional Guidance

Uniform noising (single-clock) creates a trade-off: lower $t^*$ preserves mask fidelity but suppresses background dynamics, while higher $t^*$ enables realism but forfeits motion control (Figure 3). TTM circumvents this by dual-clock denoising. Pixels in the mask are initialized with lower noise ($t_\mathrm{strong}$), enforcing tight adherence, while others start at higher noise ($t_\mathrm{weak}$), fostering plausible background dynamics. During early denoising steps, masked regions are repeatedly overridden with the corresponding noisy reference, while backgrounds freely denoise. After synchronization ($t=t_\mathrm{strong}$), regular denoising resumes for spatial coherence.

Figure 1: Contrast of region-dependent denoising strategies. SDEdit overconstrains the background at low noise and drifts at high noise; RePaint introduces region duplication artifacts; TTM’s dual-clock approach achieves both fidelity and realism.

The update rule per iteration is:

$$x_{t-1} = (1 - M)\odot \hat{x}_{t-1}(x_t,t,I) + M \odot x^w_{t-1}$$

where $x^w_{t-1}$ is the reference warped video noised to $t{-}1$, and $\hat{x}_{t-1}$ is the model prediction.

Joint Control of Motion and Appearance

Unique to TTM, full-frame conditioning on warped references enables per-pixel joint control of both motion and appearance. Unlike motion prompt-conditioned systems, users can specify appearance edits (e.g., color shifts, object insertions) frame-by-frame within the reference and enforce them in the generated video.

Figure 3: TTM supports combined motion and appearance control, with fine-grained guidance enforced across both spatial and temporal domains.

Experimental Evaluation

Object Motion Control: MC-Bench

TTM is extensively benchmarked against both training-free and training-based baselines—DragAnything, MotionPro, Go-with-the-Flow (GWTF), SG-I2V—using the MC-Bench protocol on SVD (1.5B params) and CogVideoX (5B params) backbones.

On SVD, TTM consistently achieves lowest object tracking errors (CoTracker CTD), high subject and background consistency, and eliminates background co-motion (BG–Obj CTD) compared to SG-I2V, which suffers from unintentional camera motion. On CogVideoX, TTM outperforms GWTF by a margin of 33% in MSE and 15.5% in FID on challenging camera motion benchmarks.

Figure 5: Qualitative comparison on MC-Bench. Training-based and training-free methods introduce artifacts (red), while TTM delivers accurate, clean object placement.

Figure 6: On large/discontinuous motion (right), baselines exhibit severe distortions; TTM closely follows prescribed object trajectories across different I2V backbones.

Camera Motion Control and Appearance Editing

When applied to camera motion (DL3DV), TTM constructs the warped video using monocular depth and per-frame reprojected views. Compared to GWTF, TTM demonstrates superior camera adherence (lower pixel/flow MSE, better SSIM/LPIPS), with no training or backbone changes.

Appealingly, TTM is the only approach supporting explicit joint editing of motion and appearance. For example, it enables a chameleon to traverse a user-path while changing color per user-painted reference, or supports seamless object insertions and deformations within the generated sequence.

Figure 7: For camera control, GWTF drifts, while TTM accurately follows target paths, leveraging warped depth-based references for spatial-temporal consistency.

Figure 4: Joint motion-appearance examples. TTM can enforce user-guided color transitions (top), natural object insertions with reflection (middle), and appearance-coherent shape deformations (bottom).

Implementation, Ablation, and Scaling

TTM is plug-and-play, architecture agnostic, and matches standard video diffusion runtime—the only additional requirement is constructing the user-provided animation and mask. The dual-clock schedule parameters $(t_{\text{weak}}, t_{\text{strong}})$ require tuning but generalize across diverse models (SVD, CogVideoX, WAN). Ablations confirm single-clock and RePaint-style blending are suboptimal: single-clock settings either sacrifice motion control or dynamics, while RePaint overfits to motion at the expense of visual plausibility.

The method is robust to imperfect masks and does not require fine-detailed or perfectly accurate user input, further enhancing its practical viability.

Limitations and Future Directions

TTM’s limitations include dependence on explicit object masks for motion specification, inherent anchoring only to content present in the first frame, and necessity for dual-clock tuning. The current scheme assumes two region types; generalization to varying region hierarchy, soft masks, or smoothly varying spatial-temporal noise schedules is open for exploration. Another limitation is that objects introduced after the first frame lack identity anchoring; robust extension in the occlusion/completion regime remains a future challenge. Finally, sophisticated appearance edits, articulated or long-horizon motion control, and advanced user-programming interfaces for animation definition represent promising directions enabled by the TTM foundation.

Conclusion

Time-to-Move introduces a robust, training-free solution for motion and appearance control in I2V diffusion models, imposing plug-and-play motion-conditioning without sacrificing appearance consistency or efficiency. The dual-clock denoising principle is empirically validated to achieve state-of-the-art results across diverse backbones and benchmarks, bridging the gap between interactive animation prototyping and high-fidelity video synthesis. TTM’s separation of spatially conditioned noise schedules is a practical step towards programmable, controllable video generation, likely to inform further research in region-wise or hierarchical video editing and synthesis.