Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation

Abstract: Real-time text-driven joint audio-video avatar generation requires jointly synthesizing portrait video and speech with high fidelity and precise synchronization, yet existing audio-visual diffusion models remain too slow for interactive use and often degrade noticeably after aggressive acceleration. We present Hallo-Live, a streaming framework for joint audio-visual avatar generation that combines asynchronous dual-stream diffusion with human-centric preference-guided distillation. To reduce articulation lag in causal generation, we introduce Future-Expanding Attention, which allows each video block to access synchronous audio together with a short horizon of future phonetic cues. To mitigate the quality loss of few-step distillation, we further propose Human-Centric Preference-Guided DMD (HP-DMD), which reweights training samples using rewards from visual fidelity, speech naturalness, and audio-visual synchronization. On two NVIDIA H200 GPUs, Hallo-Live runs at 20.38 FPS with 0.94 seconds latency, yielding 16.0x higher throughput and 99.3x lower latency than the teacher model Ovi. Despite this speedup, it retains strong generation quality, reaching comparable VideoAlign overall score and Sync Confidence score while outperforming other accelerated baselines in the overall quality-efficiency trade-off. Qualitative results further show robust generalization across photorealistic, multi-speaker, and stylized scenarios. To the best of our knowledge, Hallo-Live is the first framework to combine streaming dual-stream diffusion with preference-guided distillation for real-time, text-driven audio-visual generation.

• The paper introduces an asynchronous dual-stream diffusion architecture with Future-Expanding Attention to enhance lip-syncing in real-time streaming.
• The paper integrates Human-Centric Preference-Guided DMD using multimodal rewards to boost both visual fidelity and acoustic naturalness.
• The paper demonstrates a 16× throughput increase with near-teacher quality, effectively addressing real-time streaming and synchronization challenges.

Real-Time Streaming Joint Audio-Video Avatar Generation with Hallo-Live

Introduction and Problem Setting

Joint audio-video avatar generation—the task of synthesizing synchronized speech and video from textual prompts—poses significant technical challenges for achieving high fidelity and real-time performance. Existing diffusion-based multimodal models for this task are often bottlenecked by inference latency and synchronization artifacts, limiting their deployment in interactive applications. "Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation" (2604.23632) introduces a comprehensive framework to overcome these barriers through two principal technical innovations: (1) an asynchronous dual-stream diffusion architecture featuring Future-Expanding Attention for look-ahead phonetic anticipation under causal constraints, and (2) a Human-Centric Preference-Guided Distribution Matching Distillation (HP-DMD) method that incorporates multimodal reward schemes to bias distillation toward regions of the teacher’s output manifold that align with human-perceived quality and synchronization.

Figure 1: Overview of the Hallo-Live framework, spanning adaptation of a dual-stream DiT with future-expanding masks, streaming self-rollout with KV cache, and dual-stream causal fusion blocks.

Technical Approach

Asynchronous Dual-Stream Diffusion with Future-Expanding Attention

The Hallo-Live architecture builds upon a dual-stream Diffusion Transformer (DiT) backbone in which video and audio modalities are processed in parallel but allowed targeted asynchronous information exchange. The core limitation in causal streaming is the lack of short-horizon future access for the video branch: lip motions naturally precede or anticipate audible phonemes, but standard block-causal attention restricts video tokens to only current and past audio context, resulting in articulation lag.

Hallo-Live introduces Future-Expanding Attention, which extends the cross-modal receptive field for the video branch to include a tunable look-ahead window into the audio stream, thus providing necessary anticipatory cues for naturalistic synchronization.

Figure 2: Comparison between strict block-causal attention (a) and future-expanding attention (b), with sliding windows and overlapping context retention.

Training and inference leverage a cross-modal future-expanding block-causal mask so that at each streaming step, the video attends to current and a fixed number of future audio tokens. Notably, the provisional (future) audio block is never committed as output—preserving strict streaming causality—while still being used for anticipatory conditioning.

Figure 3: Visualization of cross-modal masks, contrasting strict (inaccessible) and future-expanding (selective look-ahead) patterns.

Human-Centric Preference-Guided DMD

Conventional Distribution Matching Distillation (DMD) aligns the student’s generative distribution with that of a pre-trained teacher. However, over-aggressive acceleration with vanilla DMD often incurs mean-seeking artifacts and degradation in human-centric metrics: loss of fine-grained texture, speech naturalness, and synchrony.

Hallo-Live proposes HP-DMD, wherein sample-wise distillation gradients are reweighted using multimodal rewards:

• Visual fidelity: Assessed by VideoAlign metrics.
• Acoustic naturalness: Assessed by AudioBox.
• Lip-audio synchronization: Assessed by SyncNet-based scoring.

Each sample’s reward vector is batch-standardized and combined via exponential weighting (controlled by coefficient $\beta$) to dynamically emphasize high-quality, high-sync training examples. This reward-tilted distribution enables the student to surpass average-level teacher outputs on prioritized metrics and is shown empirically to yield state-of-the-art trade-offs in efficiency and quality.

Experimental Results and Analysis

Real-Time Efficiency and Synchronization

Hallo-Live achieves 20.38 FPS with 0.94 seconds end-to-end streaming latency on dual NVIDIA H200 GPUs—a throughput increase of $16\times$ and latency reduction of $99.3\times$ compared to the Ovi teacher [low2025ovi]. Notably, this is realized without substantial loss in generation quality, synchronization, or speech intelligibility.

Figure 4: Comparative benchmarking against state-of-the-art, with Hallo-Live achieving the best efficiency-quality trade-off.

VideoAlign overall, Sync Confidence, and AudioBox scores remain competitive with much slower teacher models. Human-centric portrait fidelity (assessed via VBench-2.0) confirms near-parity with the teacher in anatomical, identity, and clothing realism.

Generalization Capabilities and Prompt Coverage

Qualitative and prompt-based analyses demonstrate robust generation under a wide range of compositional, multi-speaker, and stylized prompts—affirming the latent model’s capability to respond to semantically diverse conditioning.

Figure 5: Diverse prompt-based generations, including spatial framing, speaker multiplicity, and cartoon stylization.

Contribution of Future-Expanding Attention

Ablations varying the attention window reveal a monotonic Sync Confidence increase with larger look-ahead, saturating beyond $W=15$ video frames; this experimentally calibrates the anticipation-horizon most relevant for lip-syncing, avoiding unnecessary future leakage.

Figure 6: Sync Confidence versus attention window size, exhibiting saturation of improvements past a moderate look-ahead.

Impact of Preference Guidance

Ablation on the reward coefficients in HP-DMD across reward dimensions (Sync, VideoAlign, AudioBox) and their combinations evidences that multi-reward optimization yields balanced gains, while modality-isolated rewards benefit only their respective targets with minimal cross-modal transfer. Importantly, over-weighting (i.e., high $\beta$) sharply degrades overall fidelity due to reward hacking—optimal performance universally occurs at $\beta=2$ for non-pathological reward alignment.

Figure 7: Qualitative improvements due to reward-guided distillation, with clear visual and temporal alignment gains versus the DMD baseline.

Theoretical and Practical Implications

Hallo-Live demonstrates that effective real-time, high-fidelity text-driven audio-video generation is feasible under streaming constraints without substantial architectural or data scaling—conditional upon (a) explicit anticipation modeling in cross-modal attention, and (b) human-centric reward-aware distillation objectives. The approach illustrates practical integration of distribution matching with RL-inspired reward shaping in a multimodal generative regime, revealing nuanced interplay between synchronization, visual, and acoustic axes.

The asynchronous dual-stream setup and preference-weighted distillation paradigm can extend to broader multimodal generation tasks where real-time requirements, cross-modal synchronization, and perceptual trade-offs are paramount—such as conversational agents, AR/VR avatars, and low-latency telepresence.

Future Research Directions

Potential research trajectories include:

• Extending streaming cross-modal anticipation to more complex dependencies (e.g., facial expression, gesture, gaze).
• Fine-grained control over avatar behaviors and styles via richer prompting or modular reward design.
• Hardware-efficient distillation for deployment in resource-constrained environments.
• Leveraging human-in-the-loop preference data to dynamically refine reward models and avoid reward hacking.

Conclusion

Hallo-Live establishes a new state-of-the-art in real-time, high-quality text-driven joint audio-video avatar generation, bridging the historical gap between efficiency and perceptual synchronization. It achieves aggressive acceleration through Future-Expanding Attention and multi-reward HP-DMD while retaining nearly all teacher-level fidelity. The approach’s blend of causal anticipation and multimodal preference alignment is likely to inform future work in streaming multimodal generation, especially where low latency and human-interpretable quality are indispensable.