Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrieval

Abstract: Audio-driven facial animation is essential for immersive digital interaction, yet existing frameworks fail to reconcile real-time streaming with high-fidelity personalization. Current methods often rely on latency-inducing audio look-ahead, or require high user compliance to pre-encode static embeddings that fails to capture dynamic idiosyncrasies. We present an end-to-end causal framework for personalizing causal facial motion generation via dynamic multi-modal style retrieval, enabling ultra-low latency while uniquely leveraging unstructured style references. We introduce two key innovations: (1) a temporal hierarchical motion representation that captures global temporal context and high-frequency details while maintaining decoding causality, and (2) a multi-modal style retriever that jointly queries audio and motion to dynamically extract stylistic priors without breaking causality. This mechanism allows for scalable personalization with total flexibility regarding the number and contents of templates. By integrating these components into a causal autoregressive architecture, our method significantly outperforms state-of-the-art approaches in lip-sync accuracy, identity consistency, and perceived realism, supported by extensive quantitative evaluations and user studies.

• The paper introduces a strictly causal framework that synthesizes personalized, audio-driven facial motions in real time without look-ahead.
• The hierarchical temporal motion codec employs multi-scale discrete tokens to capture both global head movements and fine lip details, ensuring low latency.
• Dynamic multi-modal retrieval leverages historical audio and motion templates to extract style priors, outperforming baselines in synchronization and identity consistency.

Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrieval: An Expert Review

Overview and Motivation

Audio-driven facial animation plays a pivotal role in AR/VR telepresence and digital embodiment. Existing solutions either incur high latency due to audio look-ahead or diffusion-based iterative inference, or they compromise personalization fidelity relying on static or unimodal identity priors. The work "Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrieval" (Fallingwater) (2604.23692) introduces an end-to-end causal framework that integrates a hierarchical temporal motion codec and a dynamic multi-modal style retriever. This enables real-time, personalized, and highly expressive facial motion generation from streaming audio, conditioned on arbitrary sets of unstructured style references.

Figure 1: System overview of Fallingwater, detailing both the hierarchical motion codec and the motion generation process using a style retriever across unstructured template libraries.

The core contributions are two-fold: (1) a strictly causal temporal hierarchical auto-encoder producing multi-scale discrete motion codes without temporal look-ahead, and (2) a causal multi-modal retriever that dynamically queries both historical audio and motion to extract personalized priors from arbitrary reference libraries.

Hierarchical Temporal Motion Codec

A central challenge in speech-driven facial animation is capturing both low-frequency global dynamics (e.g., head pose, eyebrow gestures) and high-frequency local details (e.g., lip articulation) in a streaming, causal fashion. Fallingwater addresses this through a novel hierarchical quantization codec. A residual quantization scheme is employed, with Binary Spherical Quantization (BSQ) yielding discrete tokens at multiple temporal resolutions, progressively encoding global-to-local motion signals.

The encoder operates non-causally (leveraging future motion for robust codebook learning), while the decoder is strictly causal, using only past and current inputs during streaming inference. Critical to this process is the use of causal floor-indexed nearest interpolation for token upsampling, ensuring no future leakage during real-time decoding.

Streaming, Autoregressive Motion Generation

The hierarchical tokens are autoregressively predicted frame-by-frame by a transformer-based generator. Fallingwater's token generation proceeds in an interleaved coarse-to-fine regime: coarse tokens anchoring global motion are synthesized as soon as corresponding audio arrives, while finer tokens are generated only for frames requiring greater detail (Figure 2). This approach minimizes latency and maintains multi-scale expressivity across all facial regions.

Figure 2: Streaming motion generation predicts coarse-to-fine hierarchical tokens in an interleaved order, supporting low-latency and detailed expression.

Dynamic Multi-modal Style Retrieval

Personalization in audio-driven animation has historically suffered from insufficient or rigid conditioning. Fallingwater radically improves this through a dynamic multi-modal retriever: at each generation step, it queries not only the current and recent audio, but also the streaming history of generated motion. By searching an unstructured library of previous motion templates for stylistically similar content, the retriever supplies high-fidelity, in-context style priors irrespective of template length or specific segmentation.

The retriever avoids common exposure bias by employing a re-query training strategy: during training, the retrieval condition periodically switches between ground truth and model predictions, promoting robustness to generative drift during inference.

(Figure 1) (relevant for retriever architecture; see above)

Comprehensive Objective Formulation and Training

The codec is trained with a hybrid loss that combines spatial $L_1$ reconstruction, temporal velocity and jitter regularization (for smoothness), and entropy maximization (preventing codebook collapse). The streaming generator is trained with cross-entropy loss on discrete tokens alongside a style prior matching objective.

Evaluation metrics are tightly aligned with the real-world requirements of audio-driven facial animation: synchronization (audio-motion offset), ID-consistent Fréchet distances for both expression and pose, and learned similarity scores in identity-specific embedding spaces. Dedicated binary cross-entropy and InfoNCE objectives (Figure 3) drive the metric heads for quantitative analysis.

Figure 3: Custom evaluation metric training objectives for synchronization and identity consistency based on cross-entropy and InfoNCE, respectively.

Experimental Results

Quantitative Results

Fallingwater outperforms strong baselines (DiffPoseTalk, ARTalk, AudioRTA, MemoryTalker) in all key metrics: Sync Score (audio-motion alignment), Similarity Score (ID style match), and ID-specific Fréchet expression and pose distances. Noteworthy is the claim of true streaming (zero-lookahead), exceeding previous methods that require audio buffering or are insufficiently personalized.

Qualitative and User Study Analysis

In phoneme articulation and global head movement, the method generates output more closely matching ground truth and real-world variety than all tested baselines.

Figure 4: Qualitative phoneme articulation comparison, showing superior alignment of generated lip shapes to ground truth.

Figure 5: Qualitative head pose comparison: Fallingwater exhibits broader, more realistic motion than competing systems.

A user study corroborates numerical findings, with participants consistently preferring Fallingwater across lip sync, ID style, and naturalness metrics.

Ablation and Analysis

Exclusion of the hierarchical codec, retriever, or re-query strategies yields measurable degradation, confirming each component's necessity for state-of-the-art performance (Figure 6). Multi-modal retrieval (audio + motion) demonstrably outperforms unimodal alternatives, and increasing the template pool size flexibly improves stylization without retraining.

Figure 6: Ablation study reveals the critical effect of multi-modal retrieval and library size on synchronization and personalization.

Practical and Theoretical Implications

By fully dispensing with look-ahead and supporting arbitrary unstructured style sets, Fallingwater sets a new benchmark for real-time, identity-stylized telepresence. The hierarchical codec and interleaved generation regime may generalize to other sequence problems where streaming, conditional generation is vital. Furthermore, the multi-modal retrieval paradigm bypasses the limitations of static identity embeddings, enabling rapid personalization with minimal calibration.

The architecture’s modularity (codec + retriever + transformer) facilitates future research into richer prior modeling (e.g., generative template augmentation), broader context integration (full-body gestures), or extension to non-speech-driven behavior synthesis.

Conclusion

Fallingwater delivers a rigorously causal, highly personalized, and expressively rich solution to audio-driven facial motion synthesis. The innovations in hierarchical motion encoding, multi-modal style retrieval, and interleaved generation not only eliminate latency bottlenecks but also set a robust framework for future research in causal, personalized avatar animation and real-time telepresence. The ablation results, strong numerical improvements, and clear user preference underscore the reliability and flexibility of this approach for deployment in emerging AR/VR applications.