Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length (2512.04677v1)

Abstract: Existing diffusion-based video generation methods are fundamentally constrained by sequential computation and long-horizon inconsistency, limiting their practical adoption in real-time, streaming audio-driven avatar synthesis. We present Live Avatar, an algorithm-system co-designed framework that enables efficient, high-fidelity, and infinite-length avatar generation using a 14-billion-parameter diffusion model. Our approach introduces Timestep-forcing Pipeline Parallelism (TPP), a distributed inference paradigm that pipelines denoising steps across multiple GPUs, effectively breaking the autoregressive bottleneck and ensuring stable, low-latency real-time streaming. To further enhance temporal consistency and mitigate identity drift and color artifacts, we propose the Rolling Sink Frame Mechanism (RSFM), which maintains sequence fidelity by dynamically recalibrating appearance using a cached reference image. Additionally, we leverage Self-Forcing Distribution Matching Distillation to facilitate causal, streamable adaptation of large-scale models without sacrificing visual quality. Live Avatar demonstrates state-of-the-art performance, reaching 20 FPS end-to-end generation on 5 H800 GPUs, and, to the best of our knowledge, is the first to achieve practical, real-time, high-fidelity avatar generation at this scale. Our work establishes a new paradigm for deploying advanced diffusion models in industrial long-form video synthesis applications.

• The paper introduces a system for real-time, infinite-length avatar video synthesis using distributed diffusion and temporal consistency.
• It employs Self-Forcing DMD, Timestep-Forcing Pipeline Parallelism, and a Rolling Sink Frame Mechanism to enhance stability and throughput.
• Results demonstrate superior frame quality and synchronization over extended sequences, outperforming prior state-of-the-art methods.

Live Avatar: Scalable Real-Time Audio-Driven Avatars via Distributed Diffusion and Temporal Consistency

Introduction and Motivation

Live Avatar targets the synthesis of photorealistic, audio-driven avatar video with support for real-time streaming and infinite-length, temporally consistent generation on large-scale models. The work directly addresses two persistent bottlenecks in avatar video generation: (1) the inherent sequential and computationally intensive nature of diffusion models, which impedes real-time and streaming deployment at scale; (2) long-horizon identity and appearance drift, leading to degradation in video quality for extended use cases. While prior art has demonstrated strong visual quality in short clips or employed lightweight/quantized models for speed, Live Avatar is positioned as a system-level and algorithmic advance enabling high-fidelity, real-time, endless avatar synthesis using a 14B parameter diffusion backbone.

Live Avatar System Components

Self-Forcing Distribution Matching Distillation

The training pipeline adopts a two-stage paradigm:

1. Diffusion Forcing Pretraining: Block-wise noise scheduling and causal local attention masks are used to pretrain robust frame representations.
2. Self-Forcing DMD: Knowledge from a bidirectional, non-causal teacher is distilled into a causal few-step student generator using online distribution-matching, optimized with perturbation of the historical KV cache (Figure 1).

   Figure 1: Live Avatar training framework—Stage 1 (block-wise causal pretraining) and Stage 2 (Self-Forcing DMD with KV cache perturbation for distillation).

This approach ensures causality for deployment on streaming pipelines and robustness to temporal error accumulation. KV cache noise injection (History Corrupt) further regularizes long-horizon rollouts, enhancing stability by forcing the model to separate motion dynamics from static identity features.

Timestep-Forcing Pipeline Parallelism (TPP)

Inference efficiency is achieved through TPP, a novel distributed paradigm that spatially pipelines denoising steps across multiple GPUs. Each device is assigned a designated diffusion timestep; all GPUs denoise distinct frame blocks in parallel, breaking the sequential sampling bottleneck (Figure 2).

Figure 2: Timestep-forcing Pipeline Parallelism—Each GPU denoises at a fixed timestep, enabling fully pipelined, asynchronous throughput.

Compared to sequential or naïve sequence-parallel strategies, this enables >20 FPS end-to-end generation at 14B model scale using just 5× H800 GPUs. The throughput scales with the slowest single-timestep compute rather than total diffusion steps, and rolling per-device KV caches obviate the need for inter-GPU KV communication, reducing latency.

Rolling Sink Frame Mechanism (RSFM)

For stable, infinite-horizon generation, Live Avatar introduces RSFM, which unifies two innovations: Adaptive Attention Sink (AAS) and Rolling RoPE.

• AAS: The reference (sink) frame for identity is dynamically replaced by the model's own first output, keeping the identity anchor within the generative distribution, and mitigating drift in color/exposure.
• Rolling RoPE: The RoPE index (positional encoding) of the sink frame is dynamically shifted (Figure 3), aligning inference with training-time conditioning so that the relative position of the identity anchor and target frames remains stable throughout autoregressive rollout.

This combination provides defense against both inference-mode drift (due to misaligned attention positions) and incremental distribution drift across long sequences.

Figure 3: Rolling-RoPE mechanism—Dynamic RoPE index adjustment preserves stable positional relationships for the sink frame during long-horizon rollout.

Inference and Parallelization Analysis

TPP is implemented in two phases: pipeline warm-up and fully synchronized denoising across devices. After the short initial latency, all GPUs operate at high utilization on distinct block-timestep pairs, transferring only the denoised latent (minimal communication). VAE decoding is offloaded to a separate GPU to remove another bottleneck.

A timing analysis (Figure 4) demonstrates the system’s ability to maintain stable, high utilization on all devices with near-perfect pipelining, and rare idle gaps.

Figure 4: Multi-GPU inference timeline highlighting computation and waiting periods for each GPU, exhibiting efficient pipelined throughput.

Qualitative ablations show that without TPP, FPS collapses; sequence parallelism is insufficient due to short effective sequence per block. Diffusion step reduction with DMD is critical to latency (see ablation data in the main text).

Long-Form and Infinite-Horizon Generation

Ablation and stress testing extensively validate the temporal robustness of the approach.

• Removing AAS or History Corrupt sharply degrades perceptual and quality metrics (ASE, IQA).
• Rolling RoPE is essential for global semantic consistency (DINOv2 metric).
• Unlike prior work capped at minutes, Live Avatar demonstrates stable identity and temporal consistency out to unprecedently long rollouts (10,000 seconds), with negligible loss of metrics over time (Figure 5).

  Figure 5: Generated frames at 10s, 100s, 1000s, and 10,000s, demonstrating consistency in long-horizon extrapolation.

Qualitative and Quantitative Results

On the GenBench and AVSpeech test sets, Live Avatar matches or outperforms state-of-the-art methods on both frame quality (ASE, IQA) and synchronization (Sync-C/Sync-D), with a decisive edge on long video metrics and runtime efficiency. While step distillation does incur modest loss in the most fine-grained dynamic synchronization, user studies confirm the generated motion is more natural and globally consistent, avoiding the overfit lip behaviors penalized by human raters (see Table 4).

Qualitative comparisons on challenging OOD benchmarks (Figure 6, Figure 7) reveal that only Live Avatar sustains high fidelity and temporal coherence in diverse subject distributions over extended sequences.

Figure 6: Qualitative comparison to SOTA, highlighting Live Avatar’s superior identity and color stability on long-form output.

Figure 7: Live Avatar maintains consistency and perceptual quality compared to OmniAvatar and Echomimic-v2.

Model, Data, and Training Protocol

Live Avatar is built upon a WanS2V architecture, trained with causal 3D VAE encoding, block-wise grouping, and carefully managed audio/text conditioning. The AVSpeech dataset is filtered for high-quality long segments, and evaluation includes an OOD synthetic benchmark (GenBench) with diverse subject types and visual compositions.

Implications and Future Directions

Live Avatar establishes a new standard for scalable, real-time, perpetual avatar synthesis atop large diffusion backbones. The algorithm-system co-design, especially TPP and RSFM, will likely generalize to other streaming and real-time video diffusion contexts, including text-driven, human-AI interactive, and embodied agent applications.

Several open directions remain:

• Improving TTFF (time-to-first-frame) via tighter integration of TPP and early-exit mechanisms to further enhance interactive responsiveness.
• Extending temporal anchoring mechanisms such as RSFM to more complex, dynamic scenes with moving bodies and changing backgrounds.
• Reducing dependency on causal sink anchoring for unconstrained, open-domain generative tasks, possibly through self-supervised consistency or adversarial alignment.

Conclusion

Live Avatar delivers scalable, real-time, infinite-length audio-driven avatar generation by reconciling large-scale diffusion modeling with stringent application-level constraints. The combination of distillation for fast causal inference, distributed pipeline parallelism for throughput, and dynamic anchoring for consistency comprehensively addresses systemic limitations of prior work, opening the path to practical, interactive digital human solutions.