EchoAvatar: Real-time Generative Avatar Animation from Audio Streams

Abstract: Real-time synthesis of high-fidelity 3D character motion from audio is a pivotal component for next-generation interactive avatars and virtual assistants. However, most existing approaches are limited to offline processing of complete audio sequences or are constrained to specific domains, rarely handling both speech and music effectively. In this paper, we introduce a novel framework designed to generate continuous, coherent full-body motion from streaming speech and music with low latency. Central to our approach is a unified streaming architecture capable of synthesizing continuous motion from incremental audio inputs. We employ a robust training strategy that enforces strong audio dependency, allowing the model to seamlessly generalize across conversational speech and rhythmic music without requiring explicit domain labels or mode switching. Additionally, we explored Reinforcement Learning to refine the quality of online generation. Furthermore, we bridge reactive animation with intent-driven behavior via a tool-call interface that allows upstream LLMs to inject explicit semantic control. By combining this controllability with stream audio-driven synthesis, our framework serves as a plug-and-play solution for transforming voice agents into interactive humanoid avatars. Extensive experiments demonstrate that our method outperforms state-of-the-art realtime baselines in motion quality and synchronization while maintaining the flexibility required for live deployment. Our code, pre-trained models, and videos are available at https://robinwitch.github.io/EchoAvatar-Page.

• The paper presents a unified framework that enables real-time 3D avatar animation directly from audio using an attention-based causal motion tokenizer.
• It employs hierarchical token corruption, dual-path temporal aggregation, and residual vector quantization to streamline motion synthesis and reduce physical artifacts.
• Reinforcement learning methods (GRPO and DPO) further optimize human preference and rhythmic synchronization, validating its robust cross-domain generalization.

EchoAvatar: Real-Time Generative Avatar Animation from Audio

Technical Contributions and Unified Streaming Architecture

EchoAvatar proposes a unified framework for streaming, low-latency generation of coherent 3D avatar animation directly from audio input, bridging core limitations in existing methodologies—namely, offline, domain-specific paradigms, and restricted real-time responsiveness. Central to the system is an attention-based causal motion tokenizer, which discretizes motion trajectories without future-context dependency, enabling instantaneous inference from incremental audio streams. The tokenizer leverages causal attention masks and dual-path temporal aggregation, combined with anatomically partitioned codebooks and explicit Forward Kinematics (FK) supervision to mitigate physical artifacts such as foot-sliding and enhance upper/lower body articulation (Figure 1).

Figure 1: Architecture of the Attention-based Causal Motion Tokenizer with Residual Vector Quantization.

Residual Vector Quantization (RVQ) is employed to hierarchically compress motion, facilitating both high-fidelity reconstruction and efficient downstream generation. The system's architecture (Figure 2) supports real-time streaming input, example-based control (for stylistic modulation), and explicit semantic tool-call interface for integration with upstream LLMs.

Figure 2: Structure of the EchoAvatar motion generation model, illustrating streaming audio-to-motion synthesis and example-based style control.

Hierarchical Token Corruption and Cross-Domain Generalization

EchoAvatar identifies and resolves the conditional collapse pathology—where auto-regressive motion priors overwhelm audio conditioning in task-unified training—by introducing hierarchical token corruption. This mechanism randomly perturbs motion tokens at selected RVQ layer depths during training, enforcing reliance on acoustic input and encouraging error recovery from local sampling drift. The strategy respects RVQ’s residual hierarchy, ensuring realistic artifact modeling and robust augmentation.

Empirically, hierarchical token corruption is demonstrated as essential: its ablation leads to models that ignore input audio and generate erratic, domain-inappropriate dance-like motions even during speech (Figure 3). Reinstating corruption reinstates audio-motion correspondence and successfully enables domain-agnostic learning.

Figure 3: Comparative results showing that hierarchical token corruption is critical for robust audio-motion correspondence.

Further, EchoAvatar’s unified training across gesture and dance datasets reveals strong cross-modal synergy. Joint training facilitates emergent stylistic transfer, with gesture synthesis benefiting from improved rhythmic sensitivity derived from dance data, and dance displaying greater semantic adaptation to audio characteristics. Notably, lively, dance-like gestures emerge for energetic speech inputs, evidencing semantic generalization beyond dataset boundaries (Figure 4).

Figure 4: Jointly trained model produces domain-adaptive, energetically expressive motions responding to cheerful audio.

Reinforcement Learning Alignment: GRPO and DPO

To further align generation fidelity with human perceptual standards, EchoAvatar incorporates Reinforcement Learning (RL) post-training using Group Relative Policy Optimization (GRPO) and Direct Preference Optimization (DPO).

• GRPO utilizes self-supervised reward models for intrinsic motion quality (via token corruption monotonicity) and cross-modal synchronization (via InfoNCE-trained contrastive audio-motion embeddings). Reward models show strong generalization and discriminative power in retrieval and ordinal ranking tasks (Figure 5).
• DPO directly optimizes policy via human preference judgments over multiple sampled variants, providing greater conservatism and naturalness for gesture-centric behavior.

Distinctively, RL alignment improves subjective preference and synchronization at the cost of generative coverage (FID degradation), confirming reward-induced mode-seeking phenomena. Domain-specific analysis shows GRPO providing greater enhancement for exaggerated, rhythm-oriented dance, whereas DPO outperforms in subtle, conversational gesture settings.

Figure 5: Motion quality reward model evaluations show strong generalization and correct ordinal ranking across corruption types.

Real-Time Deployment and System Integration

EchoAvatar is deployed as a modular streaming system, compatible with browser-based or cloud-hosted voice agents, integrating LLMs for semantic control (Figure 6). Latency profiling shows that the entire pipeline (audio encoding, motion synthesis, decoding, and IK post-processing) achieves sub-266ms chunk processing on consumer and datacenter GPUs, meeting stringent real-time interaction requirements.

Figure 6: Real-time deployment pipeline comprising user host, voice agent (LLM-driven), and Motion Generator, supporting synchronous audio-motion streaming.

A sliding window approach ensures continuous inference, and post-processing with lightweight IK avoids self-intersections for stylized avatars. Subjective evaluations utilize rigorous pairwise comparison protocols, confirming robust improvements in human likeness, beat matching, and overall preference.

Quantitative and Subjective Evaluations

Comprehensive benchmarking against state-of-the-art domain-specific baselines (MECo, EDGE) and standard datasets (ZeroEGGS, Motorica, BEAT2) demonstrates EchoAvatar’s superiority in generative fidelity (lowest FID scores), rhythmic synchronization, and perceptual alignment. Ablation studies rigorously justify every architectural and optimization component; hierarchical token corruption and joint task training are pivotal for both objective metrics and emergent generalization. RL methods show domain-dependent effects and fidelity-alignment trade-offs.

Practical and Theoretical Implications

EchoAvatar establishes a scalable foundation for general-purpose avatar embodiment in conversational and entertainment contexts, providing a plug-and-play component for multimodal interactive agents. The tool-call interface with LLMs enables seamless integration of symbolic intent, opening further avenues for neuro-symbolic AI. Theoretical analysis reveals structural logit floors and the necessity of context corruption in unified learning, with broader implications for cross-modal, multitask sequence modeling.

Practically, the system addresses streaming interaction bottlenecks and supports composability by decoupling input modalities, facilitating rapid deployment in virtual assistants and creative tools. Limitations remain: the system does not yet model gaze dynamics, non-verbal backchannels, or smoothly handle abrupt acoustic transitions, restricting holistic engagement during dyadic interaction. Domain confusion due to limited speaker diversity is observed, warranting broader training datasets and advanced acoustic-semantic conditioning.

Conclusion

EchoAvatar delivers a unified, real-time audio-to-motion framework with robust generalization across speech and music, critical architectural innovations in causal tokenization, and meticulous training strategies—including hierarchical token corruption and reinforcement learning alignment. Empirical and theoretical analyses validate the system’s efficacy and lay groundwork for future fully embodied AI agents capable of seamless, contextually adaptive interaction (2605.28272).