EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents

Abstract: We present EmbodiedHead, a speech-driven talking-head framework that equips LLMs with real-time visual avatars for conversation. A practical embodied avatar must achieve real-time generation, unified listening-speaking behavior, and high rendered visual quality simultaneously. Our framework couples the first Rectified-Flow Diffusion Transformer (DiT) for this task with a differentiable renderer, enabling diverse, high-fidelity generation in as few as four sampling steps. Prior listening-speaking methods rely on dual-stream audio, introducing an interlocutor look-ahead dependency incompatible with causal user--LLM interaction. We instead adopt a single-stream interface with explicit per-frame listening-speaking state conditioning and a Streaming Audio Scheduler, suppressing spurious mouth motion during listening while enabling seamless turn-taking. A two-stage training scheme of coefficient-space pretraining and joint image-domain refinement further closes the gap between motion-level supervision and rendered quality. Extensive experiments demonstrate state-of-the-art visual quality and motion fidelity in both speaking and listening scenarios.

• The paper presents an end-to-end framework unifying LS-state conditioning and rectified-flow diffusion for real-time, speech-driven head animation.
• It achieves state-of-the-art performance with high throughput (59 FPS full render, 900+ FPS for coefficient generation) and improved visual fidelity.
• The two-stage training and streaming audio scheduler enable seamless listening and speaking transitions while suppressing artifacts in conversational settings.

EmbodiedHead: End-to-End Real-Time Head-Embodied Avatars for Conversational LLMs

Introduction and Motivation

The paradigm shift towards natural and immersive conversational AI requires agents capable of exhibiting synchronized, visually plausible head-embodied behaviors in real time, both during listening and speaking phases. The absence of non-verbal social cues such as eye contact and head motion in conventional LLM-based systems has limited perceived social presence and engagement, motivating robust speech-driven avatar frameworks. However, past approaches relying on dual-audio streams for behavior integration fail to resolve look-ahead dependencies that are unsuited to causal, turn-based interactions typical of LLM-user conversations. Additionally, decoupling mesh generation from rendering often leads to discrepancies between reported metrics and perceived quality, further compounded by the computational overhead of diffusion models in real-time deployment.

EmbodiedHead directly addresses these architectural and practical limitations, presenting a fully end-to-end framework unifying rapid inference, unified listening-speaking behavior, and SOTA visual fidelity via a Rectified-Flow Diffusion Transformer (DiT) tightly coupled with a differentiable renderer. Noteworthy architectural decisions include a single-stream causal interface with explicit listening-speaking-state (LS-state) conditioning, a streaming scheduler for audio alignment, and a two-step training scheme for bridging coefficient-space dynamics and rendered image quality.

Figure 1: EmbodiedHead employs a single-audio, LS-state–conditioned interface for natural, unified conversational behavior, contrasted with dual-audio paradigms.

Technical Approach

Rectified-Flow DiT Backbone

The model utilizes a conditional Rectified Flow DiT for FLAME-coefficient–space trajectory prediction, preserving diffusion-model diversity while dramatically reducing necessary sampling steps (down to four), undergirded by the constant-velocity straight-path property intrinsic to rectified flow. The DiT conditions on multilevel features: historical head motion (packed with exponentially grouped context tokens), multimodal audio representations (fused via learned weights from mHuBERT-147), reference identity, per-frame LS-state, and motion-magnitude guidance. Frame-local and global information are fused via self-attention, cross-attention with FiLM-based LS-dependent modulation, and AdaLN-based condition injection.

Figure 2: The pipeline leverages Rectified-Flow DiT with multi-level conditioning, streaming scheduling, and differentiable rendering for speech-driven head animation.

Single-Stream LS-State Conditioning

A principal contribution is the explicit, per-frame LS-state, injected both as token features and through FiLM-modulated audio conditioning, circumventing the interlocutor look-ahead required in dual-stream methods. The LS-state is inferred in training via active-speaker detection and at inference causally from the streaming audio source. The Streaming Audio Scheduler dynamically aligns segments of user (microphone) and LLM audio, emitting LS-state–aligned sequences for each window in a strictly causal manner, enabling robust, boundary-free turn taking.

Two-Stage Training: Dynamics and Visual Fidelity

Initial training is performed with a flow-matching loss on FLAME motion coefficients, partitioned by semantic region (expression, jaw, eyes, rotation, translation) with targeted smoothness regularization for pose parameters. To compensate for noise and bias in monocular tracking-based coefficient supervision, a second stage jointly fine-tunes both DiT and the differentiable GAGAvatar renderer using image-based L1 and perceptual (LPIPS) losses. The rectified-flow architecture allows direct one-step endpoint generation, ensuring strictly inference-aligned, end-to-end optimization.

Experimental Results

Quantitative Analysis

EmbodiedHead establishes new SOTA across 2D (PSNR, SSIM, LPIPS) and 3D (LVE, FDD, MOD, SID) metrics in both speaking and conversational scenarios, outperforming DiffPoseTalk, ARTalk, and especially DualTalk, even under the more constrained, single-stream interface. Notably, EmbodiedHead achieves a throughput of 59 FPS for full rendering and over 900 FPS for coefficient generation on a single RTX 3090, marking a significant leap in efficiency for diffusion-based methods. The explicit LS-state module suppresses unnatural articulatory artifacts during listening while supporting high-fidelity, expressive lip and upper face dynamics.

Figure 3: Exemplars demonstrate smooth, natural, and highly expressive conversational transitions between listening and speaking.

Figure 4: Comparative visual quality against baselines on the speaking test set shows greater lip-articulation accuracy and finer detail recovery.

Figure 5: In listening-speaking scenarios, EmbodiedHead outperforms DualTalk by eliminating mouth hallucinations and improving facial expressiveness.

Ablations

Isolation of major modules confirms the criticality of LS-state input and FiLM audio modulation for modality consistency and artifact suppression. The global condition branch, especially with explicit motion-magnitude guidance, endows the network with continuous, user-controllable generation amplitude scaling, demonstrating partial disentanglement of translation and rotation for kinematic expressiveness. The two-stage training pipeline is key in closing the perceptual gap between mesh-level targets and final render fidelity.

Figure 6: Ablation results highlight the impact of each architectural and conditioning module, with explicit LS-state and image-domain supervision yielding significant qualitative gains.

Implications and Future Directions

EmbodiedHead demonstrates that rectified-flow–based diffusion, coupled with explicit conversational state modeling and tight renderer integration, can meet the core demands of immersive, real-time embodied dialogue agents—namely, temporal responsiveness, interaction consistency, and photorealism. The architecture scales to single-image avatar construction and supports efficient streaming inference, both essential for practical LLM integration and deployment at scale.

The explicit LS-state paradigm lays a foundation for future extensions towards intent-sensitive, semantically driven listening behaviors by incorporating higher-level cues (textual, prosodic, or dialogue-act features) into the conditioning stack. Furthermore, hybridization of AR and diffusion paradigms or leveraging fast neural renderers could further reduce latency below current window-based frameworks. Enhanced semantic-to-motion mapping holds promise for advancing both the naturalism and utility of agent-mediated communication.

Conclusion

EmbodiedHead sets a new state of the art for real-time, speech-driven, LLM-embodied avatars by unifying efficient rectified-flow DiT modeling, explicit behavioral-state conditioning, and a two-stage, image-supervised pipeline. These innovations resolve several standing limitations of previous approaches in causal, turn-based conversational settings, delivering measurable improvements in both objective and subjective evaluation. The approach is well-positioned as a research foundation for the next generation of embodied conversational agents.

Reference:

"EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents" (2604.17211)