Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversation (2601.00664v1)

Abstract: Talking head generation creates lifelike avatars from static portraits for virtual communication and content creation. However, current models do not yet convey the feeling of truly interactive communication, often generating one-way responses that lack emotional engagement. We identify two key challenges toward truly interactive avatars: generating motion in real-time under causal constraints and learning expressive, vibrant reactions without additional labeled data. To address these challenges, we propose Avatar Forcing, a new framework for interactive head avatar generation that models real-time user-avatar interactions through diffusion forcing. This design allows the avatar to process real-time multimodal inputs, including the user's audio and motion, with low latency for instant reactions to both verbal and non-verbal cues such as speech, nods, and laughter. Furthermore, we introduce a direct preference optimization method that leverages synthetic losing samples constructed by dropping user conditions, enabling label-free learning of expressive interaction. Experimental results demonstrate that our framework enables real-time interaction with low latency (approximately 500ms), achieving 6.8X speedup compared to the baseline, and produces reactive and expressive avatar motion, which is preferred over 80% against the baseline.

• The paper presents a novel real-time diffusion-based framework that enables causal and interactive head avatar generation.
• It integrates a dual motion encoder with a causal diffusion-forcing model to produce responsive, expressive avatar reactions with sub-second latency.
• The framework leverages label-free preference optimization to enhance motion expressiveness, achieving superior performance in human studies over state-of-the-art models.

Avatar Forcing: Diffusion-Based, Real-Time Interactive Head Avatar Generation

Introduction and Problem Setting

Avatar Forcing presents a novel framework for generating interactive, real-time head avatar videos that robustly respond to users’ verbal and non-verbal cues. Prior work in talking head generation achieved significant progress in realism and lip synchronization but failed to capture truly interactive conversational dynamics. Most existing models generate one-way audio-conditioned responses, lacking low-latency bidirectional responsiveness and expressivity, which are necessary for natural human–AI conversations. The central objectives addressed are (1) real-time multimodal, causal response to user input, and (2) learning expressive avatar reactions without requiring expensive human annotation.

System Architecture

Avatar Forcing leverages a latent motion representation combined with a causal diffusion-forcing backbone to model user-avatar interactions with sub-second response times and increased expressiveness.

Figure 1: Overall architecture of Avatar Forcing, with multimodal user/avatar encoding via Dual Motion Encoder, causal motion generation, and latent-to-video decoding.

The design incorporates:

• A Dual Motion Encoder that integrates user motion, user audio, and the target avatar’s audio via cross-attention, yielding a time-synchronized conditional representation.
• A Causal Motion Generator employing blockwise look-ahead attention within a diffusion-forcing transformer, enabling low-latency rollouts without bidirectional context (constraining to only past and limited future context).

The motion generator is further augmented by a latent autoencoder that decomposes each frame into identity and motion latent vectors, providing disentanglement conducive for both expressive synthesis and efficient optimization.

Methodology: Causal Diffusion Forcing and Preference Optimization

The backbone employs diffusion forcing in the latent motion manifold, enforcing causality by precluding dependence on future user signals beyond a configurable lookahead. Instead of waiting for long temporal windows, this design supports real-time avatar updating per input block, with a rolling key-value cache to maintain context and accelerate inference.

Figure 2: Architecture of $v_{\theta}$ with look-ahead blockwise causal mask for smooth token-block transitions.

Figure 3: Bidirectional vs. causal structure comparison. Avatar Forcing (right) enables online, low-latency generation, in contrast to INFP’s bidirectional DiT (left) requiring full window context.

A critical difficulty in interactive avatar synthesis is learning expressive reactions, as the training data—particularly for listening behaviors—tends to be less dynamic and varied. To overcome this, Avatar Forcing introduces a label-free preference optimization approach based on Direct Preference Optimization (DPO) and DiffusionDPO. Synthetic “less-preferred” (under-expressive) samples are generated by dropping user-driven conditions, while ground-truth-driven samples serve as “preferred.” The DPO-based fine-tuning encourages the model to align more closely with expressive, user-reactive motion without explicit annotation or reward modeling.

Quantitative and Qualitative Results

Extensive experiments validate Avatar Forcing’s claims on dyadic conversation datasets (RealTalk, ViCo), outperforming state-of-the-art bidirectional approaches both numerically and subjectively.

• Latency: Causal blockwise generation achieves approximately 500 ms end-to-end generation latency—6.8$\times$ speedup over the strongest baseline, enabling real-time interactivity.
• Motion Reactiveness and Richness: Substantial gains in user-avatar synchronization (rPCC-Exp: 0.003 vs. 0.035/0.054) and expressive diversity (SID: 2.44 vs. 2.34/2.78).
• Human Studies: Over 80% human preference for Avatar Forcing compared to the best baseline, attributed to both reactivity and expressiveness.
• Visual Quality and Lip Sync: Maintains comparable visual realism (FID, FVD) and lip-audio alignment (LSE-D/C) to state-of-the-art models.

  Figure 4: Variance in the L2-norm of 3DMM expressions, showing the expressive gap between speaker and listener annotations in RealTalk.

  

  Figure 5: Human preference study results, with Avatar Forcing preferred in reactiveness, richness, verbal/non-verbal alignment, and overall quality.

  

  Figure 6: Qualitative comparison on RealTalk, illustrating more dynamically reactive and facially expressive avatar behaviors with Avatar Forcing.

Ablation Studies

The importance of both user motion as model input and preference optimization through DPO is confirmed via ablations.

Figure 7: Ablation—without user motion, avatars are static; with user motion, avatars mirror affective cues such as smiling and attentive focus.

Figure 8: Ablation—DPO fine-tuning yields avatars with richer and more reactive expressive motion.

Models omitting user motion, or skipping preference-based fine-tuning, exhibited substantially reduced synchronization and expressiveness, further verifying the necessity of these architectural and training innovations.

Technical Contributions and Analysis

Causal Diffusion Forcing in Motion Latents

By reformulating generation in motion latent space as a conditional vector-field ODE, Avatar Forcing avoids the inefficiency of pixel-space autoregressive models and the latency penalty of bidirectional transformers. Blockwise causal masking with lookahead ensures both temporal smoothness and strict causality, with rolling KV caching critical for efficient online inference.

DPO-based Label-Free Expressiveness Optimization

The adaptation of DPO to diffusion-based motion generation, using synthetically generated less-preferred interactions, establishes a scalable method for preference alignment absent expensive reward annotation. This removes the requirement for explicit RLHF pipelines and renders expressive, context-dependent motion learning feasible within standard supervised pipelines.

Strong Empirical Claims

• Latency reduction to ≈$500\,\mathrm{ms}$ (vs. 2.4–3.4 s baselines).
• >80% human preference across all key dimensions.
• Superior synchronization and motion richness as measured by rPCC-Exp, SID, Var.
• No degradation in visual or lip-sync quality compared to both interactive and non-interactive SOTA talking head models.

Implications and Future Directions

Practically, Avatar Forcing is positioned as an enabling technology for virtual agents in applications requiring high-fidelity, truly interactive avatars: multiplayer gaming, education, telepresence, and next-generation conversational agents. Theoretically, the application of causal latent diffusion and preference optimization for dynamic, user-responsive behavior sets a precedent for multimodal, real-time generative models that tightly couple perception and synthesis.

Figure 9: Motion Latent Encoder overview, key to fast and identity-preserving motion disentanglement.

Figure 10: Detailed view of the latent motion generator pipeline in $v_{\theta}$.

The research opens many avenues. Extension to include broader bodily cues (e.g., gesture, posture), seamless integration with multi-agent conversational frameworks, and leveraging more granular or additional modalities (gaze, physiological signals) are natural progressions. Further advances could adapt similar preference optimization concepts to text, audio, or full-body behavior generation. Finally, the blockwise causal approach and motion latent encoding introduce architectural implications for other spatiotemporal generative domains, e.g., real-time video streaming, robotics, and AR/VR content synthesis.

Conclusion

Avatar Forcing delivers a rigorously engineered framework for real-time, expressive interactive head avatar generation, combining efficient causal diffusion-based motion synthesis with label-free preference optimization for expressiveness. Empirical achievements include marked reductions in interaction latency, significant human and quantitative preference margins over prevailing baselines, and the demonstration that preference alignment can be achieved without explicit annotation or RLHF models. The system represents an integration of advanced generative modeling, efficient inference design, and state-of-the-art preference optimization for scalable, interactive avatar-based communication (2601.00664).