Vidu S1: Real-Time Interactive Video Generation

• Vidu S1 is a real-time interactive video generation system that leverages diffusion transformers for stable, infinite-length streaming with voice guidance.
• It employs a sliding-window approach with persistent reference context, TwinCache, and RoPE repositioning to maintain visual fidelity and prevent drift.
• The system achieves 540p video output at up to 42 FPS using a TurboDiffusion and TurboServe acceleration stack, while supporting personalization via uploaded images.

Vidu S1 is a real-time interactive video generation model that supports voice control of digital characters and allows users to control video generation content at any moment through voice instructions. It is designed for infinite-length real-time video generation without blurring, drift, or visual distortion, and it outputs 540p real-time videos at up to 42 FPS on regular consumer GPUs. The model is built with TurboDiffusion and TurboServe, supports personalization from uploaded images of real people, anime, and pets, and is positioned around stable long-horizon streaming, speech-guided control, and practical real-time inference (Zhang et al., 3 Jul 2026).

1. System definition and operational scope

Vidu S1 is formulated as a real-time, voice-interactive video generation system rather than a conventional offline text-to-video generator. Its central advances are explicitly described as: speech-guided control at any moment during generation; a training-to-serving stack that supports stable, infinite-length streaming by mitigating error accumulation; and practical real-time performance at 540p and up to 42 FPS on consumer GPUs (Zhang et al., 3 Jul 2026).

The model learns joint video–audio generation in latent space. If $v^i_0$ and $a^i_0$ denote clean video and audio representations at frame $i$, then the clean joint state is $x^i_0 = [v^i_0; a^i_0]$, and a sequence is written as $x^{1:N}_0 = \{x^1_0,\dots,x^N_0\}$. The unified condition $c$ includes speech, text prompts or instructions, and reference images, while $c_{1:\infty} = \{c_1,c_2,\dots\}$ provides per-frame conditioning during streaming (Zhang et al., 3 Jul 2026).

The application domain emphasized in the reported system is single-subject, talking-head or avatar scenarios. The data pipeline filters and annotates high-quality single-person, single-shot clips from livestream/talking-head and film/TV sources, and later limitations explicitly note that scaling to multi-person, complex interactions, and dynamic backgrounds remains an open direction. This suggests that Vidu S1 is best understood as a specialized streaming avatar and controllable subject-generation system rather than a general-purpose unrestricted video world simulator (Zhang et al., 3 Jul 2026).

2. Generative architecture and training objectives

The denoiser is a diffusion transformer operating on tokenized latent states for video and audio. It processes video–audio tokens with causal or bidirectional attention masks depending on training stage, cross-attends to multimodal conditions $c$, and also attends to a fixed persistent reference context derived from the user’s first frame plus the first generated state. Rotary Position Embeddings are used, and during inference the model repositions cached keys and values as the sliding window advances. Identity preservation is anchored by the persistent reference context, which remains visible through the entire stream (Zhang et al., 3 Jul 2026).

Training is organized into three stages. In Stage 1, a bidirectional teacher is trained on full sequences with complete conditioning $c_{1:N}$. At diffusion timestep $t_j$, the teacher predicts clean states

$a^i_0$0

with training loss

$a^i_0$1

This stage establishes a prior for visual fidelity and audio–visual alignment (Zhang et al., 3 Jul 2026).

In Stage 2, the model is adapted to causal streaming generation. Attention is masked so that each target frame $a^i_0$2 can use only $a^i_0$3 and the history prefix $a^i_0$4. At diffusion timestep $a^i_0$5, the autoregressive denoiser predicts

$a^i_0$6

with loss

$a^i_0$7

The historical prefix is supplied through a hybrid of Teacher Forcing, where $a^i_0$8, and Diffusion Forcing, where $a^i_0$9, chosen per sample via Bernoulli$i$0. Teacher Forcing stabilizes synchronization and motion when the past is perfect, while Diffusion Forcing trains robustness to imperfect prefixes, which is critical for long-horizon rollout (Zhang et al., 3 Jul 2026).

In Stage 3, Vidu S1 compresses sampling to a few steps through Distribution Matching Distillation and Phased Consistency Models. The DMD term is reported as

$i$1

and the PCM regularizer as

$i$2

The final objective is a weighted combination $i$3, and the deployment configuration uses 3 steps. The paper also notes that Vidu S1 is trained with direct $i$4-prediction, rather than the $i$5- or $i$6-parameterizations common in other diffusion settings (Zhang et al., 3 Jul 2026).

3. Streaming inference and infinite-length stability mechanisms

The defining systems contribution of Vidu S1 is its streaming inference design for infinite-length generation. In streaming mode, the model uses a fixed-length sliding window that contains three components: a persistent reference context formed from the user’s first frame and the first generated state; cached historical states within the window; and the current target being denoised. The persistent context functions like an attention sink frame and is always visible, stabilizing identity, style, and global layout across arbitrarily long sequences (Zhang et al., 3 Jul 2026).

A second mechanism is RoPE repositioning. Keys and values are cached before applying Rotary Position Embeddings, and as the window slides the model reapplies RoPE to cached features according to new relative positions. This preserves positional consistency without recomputing earlier activations and ensures that all positions remain within the trained range (Zhang et al., 3 Jul 2026).

A third mechanism is TwinCache, a stage-aware dual-cache design. For each generated state, Vidu S1 maintains a noisy cache at a predefined intermediate denoising step and a clean cache at the final step. During intermediate denoising, the model attends to noisy history, which retains coarse temporal dynamics and implicitly low-pass filters high-frequency artifacts that otherwise accumulate over long autoregressive sequences. At the final step, the clean cache is attended jointly with the persistent reference anchor, recovering fine details and identity consistency. The reported interpretation is explicit: temporal propagation is carried by the noisy cache, while detail refinement is performed through the clean cache, reducing drift, blurring, and texture wash-out in open-ended generation (Zhang et al., 3 Jul 2026).

These inference mechanisms are complemented by the causal training strategy and by PCM regularization in distillation. The paper states that robust causal training with Teacher Forcing and Diffusion Forcing improves tolerance to imperfect prefixes and reduces error snowballing, while PCM counters DMD’s mode collapse risk and stabilizes camera and scene layout over long horizons. Taken together, the system’s claim of infinite-length generation without blurring, drift, or distortion is grounded in this combination of persistent reference context, sliding-window causal attention with RoPE repositioning, TwinCache, robust causal training, and DMD+PCM distillation (Zhang et al., 3 Jul 2026).

4. Voice-driven control and personalization

At inference time, the conditioning stream combines three distinct signal types: raw speech audio for lip synchronization and timing, prompt-level user instructions uttered during streaming to guide future actions, and reference image latents that anchor identity. The model therefore treats speech as both a synchronization signal and a control signal. The paper emphasizes that users can talk to drive future actions, not only lip movements (Zhang et al., 3 Jul 2026).

The training data preparation is correspondingly speech-aware. The pipeline applies speech diarization, including active speaker detection, voice activity detection, and filtering of overlapping segments. It also uses dual-path annotation, with visual attributes extracted from frames and acoustic attributes extracted from audio, to produce speech-aware chunk-level captions. This improves temporal alignment and enables the generator to learn to follow spoken instructions while remaining lip-synced (Zhang et al., 3 Jul 2026).

For deployment, the paper describes a conceptual control path of

$i$7

ASR produces text tokens from streaming audio; intent parsing converts text to action, pose, emotion templates, or free-form control vectors; control mapping injects these into $i$8; and the generator updates future frames accordingly, while raw audio continues to drive lips. The paper does not disclose per-stage latencies, but states that the end-to-end generator sustains 42 FPS at 540p. A plausible implication is that most optimization effort is concentrated on the generation and streaming layers rather than on publication of the ASR or intent stack internals (Zhang et al., 3 Jul 2026).

Personalization is based on uploaded reference imagery. Users can upload a custom image of a real person, anime character, or pet, and Vidu S1 encodes this first frame into latent tokens to create the persistent reference context. This design acts as a continuous identity and style anchor. Identity preservation is reflected in the reported HDTF score of $i$9, measured as ArcFace cosine similarity. Users can also choose different voice tones, although the paper does not detail the TTS or prosody models behind that capability (Zhang et al., 3 Jul 2026).

5. Runtime system, acceleration stack, and empirical evaluation

Vidu S1’s real-time behavior is supported by the combination of TurboDiffusion-style few-step generation and the TurboServe runtime stack. Few-step sampling is achieved by DMD plus PCM distillation of the causal teacher into a 3-step generator. On the serving side, the implementation integrates attention acceleration, linear-layer quantization, kernel fusion, CUDA Graph capture and replay, and multi-GPU parallelism (Zhang et al., 3 Jul 2026).

The attention acceleration components are named explicitly: SageAttention, SpargeAttention, and Sparse-Linear Attention. Linear layers use custom per-block W8A8 quantized GEMM in CUDA. Kernel fusion employs Triton/CUDA fusions such as RMSNorm plus elementwise operations. CUDA Graph captures the stable denoising subgraph once and replays it at each frame or step to amortize host launch overhead. Multi-GPU parallelism follows Ulysses-style context parallelism, splitting the sequence across GPUs in the forward pass and using all-to-all collectives to repartition between sequence-parallel and head-parallel layouts around attention blocks (Zhang et al., 3 Jul 2026).

With these optimizations, the reported deployment target is 540p, specifically $x^i_0 = [v^i_0; a^i_0]$0, at up to 42 FPS on RTX 5090 GPUs in a 3-step configuration. The paper explicitly states that this exceeds the 30 FPS threshold for real-time applications (Zhang et al., 3 Jul 2026).

On HDTF, the reported benchmark scores are best among reported systems for the listed metrics: $x^i_0 = [v^i_0; a^i_0]$1, $x^i_0 = [v^i_0; a^i_0]$2, and $x^i_0 = [v^i_0; a^i_0]$3. The paper compares Vidu S1 with HeyGen, LemonSlice, Kling Avatar 2.0, and additional systems including LiveAvatar, OmniAvatar, StableAvatar, Wan2.2-S2V-14B, and Hallo3, while emphasizing that none combine real-time, instruction following, and top metrics like Vidu S1 (Zhang et al., 3 Jul 2026).

System	Speed / Resolution	Reported HDTF metrics
Vidu S1	540p, up to 42 FPS	CSIM 0.9192; Sync-D 7.847; DOVER 0.5660
HeyGen	25 FPS	CSIM 0.9191; Sync-D 8.037; DOVER 0.4864
LemonSlice	368×560	CSIM 0.8407; Sync-D 7.921; DOVER 0.5196
Kling Avatar 2.0	not specified in the summary block	CSIM 0.8688; Sync-D 8.158; DOVER 0.5406

The in-house Vidu-StreamBench contains 500 samples, each with a reference image, audio, and action instruction. Human pairwise preference tests versus HeyGen, LemonSlice, and Kling Avatar 2.0 reportedly show Vidu S1 consistently preferred overall. In subject controllability, Vidu S1 attains a 100% preference rate against HeyGen and LemonSlice, which the paper interprets as stronger instruction following and more natural motion and gestures (Zhang et al., 3 Jul 2026).

6. Relation to earlier Vidu and known limitations

The name “Vidu S1” is distinct from the earlier system introduced as “Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models.” That 2024 paper presents Vidu as a high-performance text-to-video generator capable of producing 1080p videos up to 16 seconds in a single generation, with a U-ViT backbone and a video autoencoder, but the paper does not use the term “S1” anywhere, nor does it describe Stage-1/Stage-2 nomenclature for that system (Bao et al., 2024).

This distinction matters because the two systems occupy different operating regimes. The 2024 Vidu emphasizes single-generation, long-form text-to-video synthesis with strong coherence, dynamism, and controllable extensions such as canny-to-video generation, video prediction, and subject-driven generation. By contrast, Vidu S1 is framed around real-time interactive streaming, voice-driven future action control, infinite-length generation, and the deployment mechanics required to sustain 540p at up to 42 FPS (Bao et al., 2024). This suggests a substantive shift from offline text-conditioned video generation toward interactive, persistent, avatar-centric video generation.

The limitations reported for Vidu S1 are also specific. The paper notes instability for singing or strong music and filters such segments during data preparation. The training data emphasize static or slow shots to reduce drift risk, so rapid head or body motion, occlusions, and complex camera moves at the same real-time rates remain open directions. The system is tailored to single-subject scenarios, and scaling to multi-speaker, noisy environments, multi-person composition, and dynamic backgrounds without sacrificing real-time rates is presented as future work. The ASR and intent-parsing stack is described conceptually but not disclosed in detail, and the paper observes that DMD alone can cause collapse, with PCM regularization mitigating that risk (Zhang et al., 3 Jul 2026).

In the present literature, Vidu S1 is therefore best characterized as a real-time, voice-interactive diffusion-transformer system for personalized, single-subject streaming video generation, distinguished by persistent reference anchoring, causal windowed inference with RoPE repositioning, TwinCache, and a deployment-oriented acceleration stack, rather than as a renamed stage of the earlier Vidu text-to-video model (Zhang et al., 3 Jul 2026).