Vidu S1: A Real-Time Interactive Video Generation Model

Abstract: We introduce Vidu S1, a real-time interactive video generation model supporting voice control of digital characters. Users can control video generation content at any moment through voice instructions. Vidu S1 supports infinite-length real-time video generation without blurring, drift, or visual distortion. Built with TurboDiffusion and TurboServe, Vidu S1 outputs 540p real-time videos at up to 42 FPS on regular consumer GPUs. Users can upload custom images of real people, anime, and pets, and choose different voice tones for personalized experiences. Experiments show that Vidu S1 achieves the best performance across all test metrics while fully meeting real-time inference requirements. A playable online demo is available at https://vidu.com/vidu-stream.

• The paper introduces a real-time autoregressive video generation framework that enables speech-driven, interactive control of digital avatars.
• It employs a three-stage training paradigm with Distribution Matching Distillation and Phased Consistency Models to ensure temporal stability and identity preservation.
• The model integrates an optimized hardware-software inference stack to achieve 42 FPS at 540p, outperforming commercial baselines in key evaluations.

Vidu S1: A Real-Time Interactive Video Generation Model

Motivation and Model Overview

Vidu S1 presents a real-time autoregressive video generation framework enabling voice-controlled, instant interaction with digital avatars. The model is architected to address distinct deficits in previous approaches, namely: lack of real-time user guidance during generation, limited support for speech-driven controls, instability and drift during long-horizon generation, and prohibitive computational cost for real-time deployment. Through integrated advances in model architecture, data pipeline, inference stacking, and deployment, Vidu S1 achieves high-resolution, low-latency, interactive video synthesis.

Figure 1: Overview of Vidu S1. Vidu S1 supports speech-guided future control, efficient infinite-duration inference, and diverse customized characters for real-time interactive video generation.

Data Processing Pipeline

High-quality, interactive video generation requires semantically aligned, temporally stable, diverse training data. Vidu S1's data pipeline executes a rigorous multi-stage process: technical prefiltering, deduplication, single-shot clipping at semantically meaningful boundaries, multimodal subject and quality filtering, and safety assurance. The system leverages both local expert models and large-scale omni models for semantic attribute detection, facilitating robust subject selection and content verification. Fine-grained speech diarization, aided by heuristic post-processing for music-speech separation, and structured dual-path visual/audio annotation yield temporally and semantically aligned conditioning for training.

Figure 2: Data filtering pipeline for raw videos, showing progressive refinement through prefiltering, single-shot clipping, subject filtering, quality and safety checks, diarization, and caption generation. The final selected clips and their captions are embedded to form training data.

Training Methodology

Vidu S1 employs a three-stage, hybrid training approach for stable, efficient, and causal video-audio generation:

1. Bidirectional Teacher Training: The base model learns to denoise video-audio sequences in a fully observable, bidirectional paradigm, optimizing for multimodal fidelity and alignment.
2. Causal Teacher Adaptation: The model is adapted to the causal, streaming regime via hybrid Teacher Forcing and Diffusion Forcing, ensuring robustness to both perfect and imperfect history during autoregressive generation.
3. Distribution Matching Distillation (DMD) + Phased Consistency Models (PCM): Efficient few-step autoregressive sampling is distilled from the causal teacher by optimizing a distribution-matching objective, regularized with perceptual consistency losses to mitigate mode collapse and ensure long-horizon stability.

The methodology leverages explicit conditioning on user-supplied modalities (reference image, speech, text) and incorporates sliding-window decoding, RoPE repositioning, and stage-aware caching (TwinCache) for efficient and temporally stable streaming inference. This enables unlimited sequence length without increased per-step latency, with robust appearance and identity continuity.

Inference and Deployment Acceleration

Vidu S1 is deployed via a jointly optimized hardware-software inference stack. Key architectural innovations include:

• Attention Kernel Acceleration: Deployment utilizes SageAttention, SpargeAttention, and SLA for 8/4-bit attention processing, enabling SOTA attention throughput without precision loss.
• Linear Layer Quantization: Custom per-block W8A8 quantized GEMM operators reduce memory and accelerate computation for linear layers, maintaining fidelity against outlier quantization artifacts.
• Kernel Fusion and CUDA Graph: Operator chains are fused into Triton/CUDA kernels, and execution graphs are captured and replayed to minimize host launch latency and boost GPU utilization.
• Ulysses-Style Context Parallelism: Multi-GPU distribution with sequence/head partitioning and all-to-all synchronization supports high-throughput real-time inference.
• TurboDiffusion and TurboServe Integration: The model leverages the optimized worker and cluster serving strategies from TurboDiffusion and TurboServe to support high-resolution (540p) interactive generation at up to 42 FPS even on commodity GPUs.

Evaluation and Results

A comprehensive evaluation protocol assesses controllability, perceptual quality, identity preservation, and audio-video synchronization. Vidu S1 introduces the Vidu-StreamBench for real-time, instruction-following benchmarks covering variable speakers, visual styles, emotional states, action commands, and durations. It is compared with HeyGen, LemonSlice, and Kling-Avatar-2.0, as well as evaluated with public metrics on HDTF.

Figure 3: Human preference evaluation of Vidu S1 versus HeyGen, LemonSlice, and Kling-Avatar-2.0 on Vidu-StreamBench.

Vidu S1 demonstrates best-in-class results:

• CSIM: 0.9192 (identity preservation)
• Sync-D: 7.847 (audio-visual sync, lower is better)
• DOVER: 0.5660 (perceptual video quality)
• Instruction Following and Real-Time Capability: Achieves 42 FPS at 540p while supporting explicit, online user guidance.

Instruction-following is a marked point of superiority: Vidu S1 is unanimously preferred by human raters for subject controllability over strong commercial baselines, and exhibits robust execution of explicit commands versus simple passive talking.

Figure 4: Qualitative comparison of instruction following and visual consistency. Green boxes: Vidu S1, which correctly executes prompts; Red boxes: competitor failures (e.g., missing gestures, background artifacts).

Practical and Theoretical Implications

Vidu S1 sets a practical precedent for real-time, speech-guided, infinitely extensible video generation at high resolution and low latency. Its methodology demonstrates that robust, long-horizon autoregressive diffusion is feasible in practice without sacrificing fidelity or control. The staged training paradigm with DMD + PCM is notable for preventing temporal drift and mode collapse, which are chronic in prior streaming video models. The technical stack—especially in quantization, kernel fusion, and multi-GPU scaling—provides a direct pathway to economical, interactive deployment.

On the theoretical side, the model's successful integration of multimodal, speech-centric conditioning, and robust open-ended generation, highlights new standards in aligning generative video models with real-world interactive contexts. TwinCache and RoPE repositioning offer new design patterns potentially generalizable to other streaming sequence models.

Future Prospects

Anticipated directions include expansion to higher-resolution (e.g., 1080p, 4K) real-time generation, domain transfer to non-avatar or open-world video, enhancement of multi-modal interaction (gesture, text, scene manipulation), and tighter integration with edge or mobile devices. Further theoretical investigation can focus on extending distribution-matching distillation for even more complex, multi-actor, multi-turn interactive scenarios. Vidu S1 establishes a baseline for real-time digital content creation frameworks that are fully user-driven, temporally extensible, and practical for wide deployment.

Conclusion

Vidu S1 delivers an integrated solution to real-time, instruction-following interactive video generation by combining data curation, hybrid teacher training, efficient diffusion distillation, and advanced inference acceleration. This model achieves unmatched scores in perceptual, synchronization, and identity metrics, with robust real-time performance, marking significant progress toward practical, user-controllable generative video avatars. Its architectural and algorithmic strategies define a clear future path for scalable, interactive video synthesis.