MotionStream: Real-Time Video Generation with Interactive Motion Controls (2511.01266v1)

Abstract: Current motion-conditioned video generation methods suffer from prohibitive latency (minutes per video) and non-causal processing that prevents real-time interaction. We present MotionStream, enabling sub-second latency with up to 29 FPS streaming generation on a single GPU. Our approach begins by augmenting a text-to-video model with motion control, which generates high-quality videos that adhere to the global text prompt and local motion guidance, but does not perform inference on the fly. As such, we distill this bidirectional teacher into a causal student through Self Forcing with Distribution Matching Distillation, enabling real-time streaming inference. Several key challenges arise when generating videos of long, potentially infinite time-horizons: (1) bridging the domain gap from training on finite length and extrapolating to infinite horizons, (2) sustaining high quality by preventing error accumulation, and (3) maintaining fast inference, without incurring growth in computational cost due to increasing context windows. A key to our approach is introducing carefully designed sliding-window causal attention, combined with attention sinks. By incorporating self-rollout with attention sinks and KV cache rolling during training, we properly simulate inference-time extrapolations with a fixed context window, enabling constant-speed generation of arbitrarily long videos. Our models achieve state-of-the-art results in motion following and video quality while being two orders of magnitude faster, uniquely enabling infinite-length streaming. With MotionStream, users can paint trajectories, control cameras, or transfer motion, and see results unfold in real-time, delivering a truly interactive experience.

• The paper presents a two-stage teacher-student framework that distills a bidirectional video diffusion model into a causal, streaming model, enabling real-time generation.
• It introduces sinusoidal embedding-based motion control that achieves 40× faster encoding for interactive applications like motion transfer and camera control.
• Quantitative results demonstrate state-of-the-art performance with up to 29 FPS and competitive PSNR/LPIPS scores, supporting infinite, high-quality video synthesis.

MotionStream: Real-Time Video Generation with Interactive Motion Controls

Introduction and Motivation

MotionStream addresses the critical limitations of prior motion-controlled video diffusion models, which are constrained by high latency, non-causal (offline) processing, and short-duration outputs. These constraints preclude real-time, interactive video generation, a capability essential for creative applications such as live motion transfer, drag-based editing, and camera control. MotionStream introduces a streaming, autoregressive video generation framework that enables sub-second latency and up to 29 FPS on a single GPU, supporting infinite-length video synthesis from a single image with interactive motion trajectory control.

Figure 1: MotionStream enables streaming, interactive video generation from a single image with track control, supporting real-time applications such as motion transfer, drag operations, and 3D camera control.

Model Architecture and Training Pipeline

MotionStream is built upon a two-stage teacher-student framework. The teacher is a bidirectional, motion-controlled video diffusion model, while the student is a distilled, causal (autoregressive) model capable of streaming inference. The teacher model is augmented with a lightweight, sinusoidal-embedding-based track head for efficient trajectory conditioning, avoiding the computational overhead of ControlNet-style architectures. The teacher is trained with a rectified flow matching loss, supporting joint text and motion guidance via classifier-free guidance (CFG).

Figure 2: The architecture and training pipeline: a bidirectional teacher model with motion control is distilled into a causal student via Self Forcing-style DMD distillation, using rolling KV cache and attention sink for stable, streaming inference.

Motion Control and Track Representation

Motion trajectories are encoded as sinusoidal embeddings with a learnable track head, which are spatially placed at downsampled grid locations. This approach is empirically superior to RGB-encoded tracks processed by a VAE, yielding both higher fidelity and 40× faster encoding, which is critical for real-time streaming.

Joint Text and Motion Guidance

MotionStream employs a hybrid guidance strategy, combining text and motion CFG. Text guidance enables natural, dynamic motion, while motion guidance enforces strict trajectory adherence. The joint guidance formula is:

$$\hat{v} = v_{\text{base}} + w_t \cdot (v(c_t, c_m) - v(\varnothing, c_m)) + w_m \cdot (v(c_t, c_m) - v(c_t, \varnothing))$$

with empirically tuned weights ($w_t=3.0$, $w_m=1.5$) to balance realism and control.

Figure 3: Quantitative ablation of guidance strategies: higher text guidance reduces trajectory accuracy but improves visual quality, while motion guidance does the opposite.

Causal Distillation and Streaming Inference

The core innovation is the distillation of the slow, bidirectional teacher into a fast, causal student using Self Forcing with Distribution Matching Distillation (DMD). The student model is trained to autoregressively generate video chunks, attending only to a fixed-size local window and a persistent "attention sink" (initial frame tokens), using a rolling KV cache. This design ensures constant-speed, infinite-horizon generation without context window growth or quality drift.

Attention Sink and Sparse Attention Patterns

Analysis of attention maps reveals that anchoring to the initial frame (attention sink) is crucial for long-term stability, preventing cumulative drift during extrapolation. The optimal configuration is a single-chunk sink and a single-chunk window, as larger windows increase error accumulation.

Figure 4: Impact of sparse attention patterns: a single sink chunk is essential for stability, while larger windows degrade performance due to error accumulation.

Figure 5: Long video extrapolation: models without attention sink exhibit drift, while MotionStream maintains stable quality throughout.

Chunk Size and Sampling Steps

A chunk size of 3 with 3 sampling steps is optimal, balancing latency, throughput, and quality. Larger chunks increase latency, while smaller chunks degrade quality.

Figure 6: Speed and quality trade-offs: chunk size and sampling steps affect latency, throughput, and LPIPS quality.

Quantitative and Qualitative Results

MotionStream achieves state-of-the-art results in motion transfer and camera control benchmarks, outperforming prior methods in both fidelity and speed. On the DAVIS and Sora datasets, the causal (distilled) model achieves 16.7 FPS at 480P and 10.4 FPS at 720P, with PSNR and LPIPS competitive with or superior to all baselines except ATI, which uses a 10× larger backbone.

Figure 7: User paper results: MotionStream models are preferred over all baselines except ATI, which uses a much larger backbone.

Figure 8: Qualitative results: MotionStream supports long video motion transfer, drag-based controls, and precise camera control with depth estimation.

Figure 9: Qualitative comparison: MotionStream reconstructs complex motions with high fidelity, outperforming GWTF and matching ATI in visual quality while maintaining better trajectory adherence.

Streaming Demo and Real-Time Applications

The streaming demo demonstrates real-time, interactive video generation with user-drawn trajectories, drag operations, and camera control. The pipeline is further accelerated by a custom Tiny VAE decoder, achieving up to 29.5 FPS with minimal quality loss.

Figure 10: Streaming demo: real-time drag, static region specification, and multi-point trajectory control are supported interactively.

Limitations and Failure Cases

MotionStream's fixed attention sink mechanism limits its ability to handle complete scene changes, as it anchors to the initial frame. Extremely rapid or physically implausible trajectories can cause artifacts or temporal inconsistencies. Backbone capacity remains a bottleneck for highly complex scenes or motions.

Figure 11: Failure cases: complex or physically implausible motions can result in deformations or loss of identity, especially in multi-object scenes.

Implications and Future Directions

MotionStream demonstrates that real-time, interactive, infinite-length video generation with precise motion control is feasible on a single GPU. The architecture's modularity allows for integration with larger backbones, improved track augmentation, and dynamic attention sinking for world modeling applications. The approach is directly applicable to creative tools, interactive content creation, and simulation environments. Future work should address dynamic scene transitions, robustness to imperfect user input, and scaling to larger, more capable backbone models.

Conclusion

MotionStream establishes a new paradigm for interactive video generation, combining efficient motion control, causal distillation, and sparse attention mechanisms to achieve real-time, infinite-horizon synthesis. The framework delivers state-of-the-art quality and speed, enabling practical deployment in creative and interactive applications. The technical innovations in track representation, joint guidance, and attention sink design are broadly applicable to future autoregressive video generation and world modeling systems.