MemoryTalker: Audio-Driven 3D Facial Animation

• MemoryTalker is a speech-driven 3D facial animation framework that generates personalized facial motion directly from raw audio without requiring identity labels or auxiliary motion inputs.
• It employs a two-stage pipeline where a multimodal motion memory first captures general facial motion patterns, which are then modulated by audio-derived style features for personalization.
• The framework decouples content and style using ASR-derived text features and mel-spectrogram cues, enabling efficient real-time applications in VR, gaming, and virtual avatar systems.

MemoryTalker is a speech-driven 3D facial animation framework that synthesizes personalized facial motion sequences from audio alone, without requiring identity labels, reference 2D videos, or reference 3D meshes at inference time. It was introduced to address a central limitation of prior personalization strategies: one-hot identity conditioning does not generalize to unseen speakers, while reference-motion-driven methods depend on auxiliary motion inputs that are impractical in real-time deployment. The framework is organized as a two-stage pipeline—“Memorizing” and “Animating”—in which a multimodal motion memory stores general facial motion patterns, and an audio-guided stylization module modulates that memory according to speaker-specific cues extracted from the input audio (Kim et al., 28 Jul 2025).

1. Problem formulation and design motivation

Speech-driven 3D facial animation aims to synthesize temporally coherent 3D face mesh motions synchronized with input speech. In the formulation used by MemoryTalker, high-quality synthesis is not limited to lip synchronization; it must also capture personal speaking style, including mouth opening amplitude, pouting extent, and subtle articulation differences across speakers (Kim et al., 28 Jul 2025).

The framework is motivated by two deficiencies in earlier personalization paradigms. First, one-hot identity conditioning, used by systems such as VOCA, FaceFormer, and CodeTalker, encodes speaker identity during training and requires selecting an identity at inference time. This setup fails for unseen speakers and cannot recover their style without labels. Second, reference-motion-driven approaches such as Imitator, Composite/Regional Movements, Mimic, and Probabilistic SD 3D Facial can adapt style from auxiliary 2D video or 3D mesh sequences, but they require additional inputs that are often unavailable in real-time applications (Kim et al., 28 Jul 2025).

MemoryTalker therefore targets a narrower but operationally important objective: arbitrary-speaker personalization from raw audio only. Its central design choice is to decouple content and style. Content is represented by ASR-derived, speaker-agnostic text features, while style is derived from mel-spectrogram features that encode prosody, rhythm, and timbre. This separation allows the system to retrieve general motion patterns from memory and then emphasize the motion types most compatible with the current speaker’s speaking style (Kim et al., 28 Jul 2025).

A common misunderstanding is to read the word “memory” as implying identity lookup or explicit user-specific memory storage. In MemoryTalker, the memory is instead a learnable bank of general facial motion slots. Personalization is achieved by modulating those slots with audio-driven style weights, not by retrieving a stored subject-specific template at inference time (Kim et al., 28 Jul 2025).

2. Representation and architectural components

The model represents facial motion as per-vertex displacements relative to a neutral registered mesh template. The facial motion at time $t$ is denoted $v^t \in \mathbb{R}^3$, corresponding to per-vertex 3D movement with respect to the neutral face. On VOCASET, the mesh topology is FLAME with 5,023 vertices at 60 fps; on BIWI, the topology has 23,370 vertices at 25 fps, with average sequence length approximately 4.67 s (Kim et al., 28 Jul 2025).

The architecture has five principal components. The audio-to-text encoder $E_{aud}$ is a pretrained HuBERT ASR encoder. After interpolation to match the video frame rate, a projection layer maps the output to memory-key channels:

$$f_{txt}^{1:T} = \psi_{\to n}(\mathrm{Interp}(E_{aud}(a))) \in \mathbb{R}^{T \times n}.$$

These features are intended to encode phonetic content in a speaker-agnostic form (Kim et al., 28 Jul 2025).

The motion encoder $E_m$ is a single linear layer that maps per-frame vertex displacements into a latent motion feature $f_m^t \in \mathbb{R}^c$. The learnable motion memory $M_m$ contains $n$ slots, each a $c$-dimensional vector:

$$M_m = \{s_m^i\}_{i=1}^n \in \mathbb{R}^{n \times c}.$$

During memorization, the model computes value-addressing weights from the motion feature using cosine similarity:

$v^t \in \mathbb{R}^3$0

and reconstructs a recalled motion feature

$v^t \in \mathbb{R}^3$1

A parallel key-addressing path uses the ASR-derived text representation:

$v^t \in \mathbb{R}^3$2

with general motion recall

$v^t \in \mathbb{R}^3$3

The paper presents this memory as a bridge between audio-text keys and motion values across speakers (Kim et al., 28 Jul 2025).

Personalization is introduced by a speaking style encoder $v^t \in \mathbb{R}^3$4 that processes mel-spectrograms $v^t \in \mathbb{R}^3$5 with 80 mel bins. According to the supplementary implementation summary, this module uses Conv1d stacks, GroupNorm, pooling, and MLP layers to produce a style feature $v^t \in \mathbb{R}^3$6. The style feature is converted into per-slot weights,

$v^t \in \mathbb{R}^3$7

which scale the motion memory slotwise:

$v^t \in \mathbb{R}^3$8

Stylized recall then becomes

$v^t \in \mathbb{R}^3$9

This multiplicative modulation is the mechanism by which MemoryTalker emphasizes motion types associated with a particular speaking style (Kim et al., 28 Jul 2025).

The temporal decoder $E_{aud}$0 is transformer-based, following the FaceFormer style of sequence decoding. It reconstructs either general motion,

$E_{aud}$1

or personalized motion,

$E_{aud}$2

The overall architecture is therefore neither a direct audio-to-mesh regressor nor a template transfer system; it is a memory-conditioned decoder in which content selects motion slots and style rescales them (Kim et al., 28 Jul 2025).

3. Two-stage training procedure

MemoryTalker is trained in two stages. The first stage, “Memorizing,” learns to store and retrieve general motion patterns independently of speaker style. The second stage, “Animating,” freezes the first-stage modules and trains only the style encoder so that audio-derived style information can modulate the memory for personalized synthesis (Kim et al., 28 Jul 2025).

In the memorizing stage, the model aligns motion-derived value addresses with text-derived key addresses. The alignment loss is

$E_{aud}$3

and the memory reconstruction loss is

$E_{aud}$4

Sequence reconstruction is supervised with mesh reconstruction loss

$E_{aud}$5

and velocity consistency loss

$E_{aud}$6

The total first-stage objective is

$E_{aud}$7

with $E_{aud}$8 (Kim et al., 28 Jul 2025).

In the animating stage, the style encoder receives mel-spectrograms and is trained with a triplet-style loss,

$E_{aud}$9

where positive and negative style features are drawn from same-speaker and different-speaker samples, respectively. The model also uses a lip-region emphasis term,

$$f_{txt}^{1:T} = \psi_{\to n}(\mathrm{Interp}(E_{aud}(a))) \in \mathbb{R}^{T \times n}.$$0

The second-stage objective is

$$f_{txt}^{1:T} = \psi_{\to n}(\mathrm{Interp}(E_{aud}(a))) \in \mathbb{R}^{T \times n}.$$1

with $$f_{txt}^{1:T} = \psi_{\to n}(\mathrm{Interp}(E_{aud}(a))) \in \mathbb{R}^{T \times n}.$$2 (Kim et al., 28 Jul 2025).

The training protocol follows FaceFormer and CodeTalker splits. On VOCASET, the data comprise 480 sequences, 12 subjects, and 255 sentences at 60 fps, with approximately 4 s per sequence. BIWI contains 40 shared sentences across 14 subjects, including neutral and emotional variants, at 25 fps. Stage 1 trains all modules except $$f_{txt}^{1:T} = \psi_{\to n}(\mathrm{Interp}(E_{aud}(a))) \in \mathbb{R}^{T \times n}.$$3 for 100 epochs with learning rate $$f_{txt}^{1:T} = \psi_{\to n}(\mathrm{Interp}(E_{aud}(a))) \in \mathbb{R}^{T \times n}.$$4 on an NVIDIA A6000 GPU. Stage 2 freezes all stage-1 modules and trains only $$f_{txt}^{1:T} = \psi_{\to n}(\mathrm{Interp}(E_{aud}(a))) \in \mathbb{R}^{T \times n}.$$5 for 100 epochs with learning rate $$f_{txt}^{1:T} = \psi_{\to n}(\mathrm{Interp}(E_{aud}(a))) \in \mathbb{R}^{T \times n}.$$6. The number of memory slots is fixed to $$f_{txt}^{1:T} = \psi_{\to n}(\mathrm{Interp}(E_{aud}(a))) \in \mathbb{R}^{T \times n}.$$7, selected by ablation among $$f_{txt}^{1:T} = \psi_{\to n}(\mathrm{Interp}(E_{aud}(a))) \in \mathbb{R}^{T \times n}.$$8 (Kim et al., 28 Jul 2025).

This two-stage decomposition suggests a deliberate bias toward content stability before style specialization. The paper’s end-to-end comparison supports that reading: on VOCASET, end-to-end training yields FVE 0.510, LVE 0.303, and FID 3.142, whereas the two-stage setup yields 0.506, 0.293, and 3.045 (Kim et al., 28 Jul 2025).

4. Evaluation protocol and empirical performance

The evaluation uses Face Vertex Error (FVE), Lip Vertex Error (LVE), Lip Dynamic Time Warping (LDTW), Fréchet Inception Distance (FID), and Lip-max. FVE is the mean Euclidean error across face vertices and frames; LVE restricts that computation to lower-face or lip vertices; LDTW measures temporal similarity of lip trajectories; FID is computed on rendered images; and Lip-max captures the average of the maximum per-frame lip-region vertex error (Kim et al., 28 Jul 2025).

On VOCASET, MemoryTalker reports FVE $$f_{txt}^{1:T} = \psi_{\to n}(\mathrm{Interp}(E_{aud}(a))) \in \mathbb{R}^{T \times n}.$$9, LVE $E_m$0, FID $E_m$1, LDTW $E_m$2, and Lip-max $E_m$3. The paper states that these results surpass FaceFormer, which records $E_m$4, and UniTalker, which records $E_m$5 in the same metric order. On BIWI, MemoryTalker reports FVE $E_m$6, LVE $E_m$7, FID $E_m$8, LDTW $E_m$9, and Lip-max $f_m^t \in \mathbb{R}^c$0, surpassing FaceFormer and CodeTalker according to the reported comparison (Kim et al., 28 Jul 2025).

The paper also isolates an instability in one-hot identity-conditioned models. For the same audio input on VOCASET, FaceFormer’s FVE varies as $f_m^t \in \mathbb{R}^c$1, CodeTalker’s as $f_m^t \in \mathbb{R}^c$2, and Imitator’s as $f_m^t \in \mathbb{R}^c$3, depending on the selected training identity. MemoryTalker produces a single result, reported as FVE 0.506, LVE 0.293, and FID 3.045, because it does not require identity labels at inference (Kim et al., 28 Jul 2025).

Ablation results attribute performance gains to both the memory stage and the style stage. On VOCASET, a baseline without memory or style obtains FVE 0.638 and LVE 0.460; adding first-stage memory alone improves these to 0.531 and 0.313; the full two-stage model reaches 0.506 and 0.293. Removing $f_m^t \in \mathbb{R}^c$4 or $f_m^t \in \mathbb{R}^c$5 yields modest degradations, for example FVE 0.514 or 0.513 instead of 0.506, and LDTW 0.435 or 0.431 instead of 0.418. A separate comparison between mel-based and ASR-derived style features finds that mel-based $f_m^t \in \mathbb{R}^c$6 performs better on VOCASET, with FVE 0.506 versus 0.518, LVE 0.293 versus 0.300, and LDTW 0.418 versus 0.437 (Kim et al., 28 Jul 2025).

The runtime profile is one of the framework’s notable practical attributes. For 1 s of audio, MemoryTalker requires 7.8 ms of inference time on an NVIDIA A6000 GPU, corresponding to approximately 120–128 fps, with 94M parameters. In the reported efficiency comparison, this is faster than FaceFormer at 38.1 ms, CodeTalker at 297.6 ms, SelfTalk at 10.1 ms, and UniTalker at 9.7 ms, while remaining substantially smaller than CodeTalker and UniTalker in parameter count (Kim et al., 28 Jul 2025).

Subjective evaluation is reported on VOCASET with 33 participants in A/B tests. MemoryTalker is preferred over FaceFormer, CodeTalker, Imitator, ScanTalk, and UniTalker across lip-sync, realism, and speaking style. For example, against FaceFormer it receives 83.9% preference in lip-sync, 85.5% in realism, and 80.6% in style; against ScanTalk the corresponding figures are 94.4%, 91.1%, and 92.8%. Against ground truth, MemoryTalker is selected 41.1% of the time for lip sync and 40.3% for realism, which the paper presents as indicating proximity to the ground-truth sequences in subjective assessment (Kim et al., 28 Jul 2025).

5. Relation to prior and subsequent systems

Relative to earlier personalized facial animation methods, MemoryTalker occupies a distinct position. It removes identity priors at inference, unlike one-hot-conditioned methods, and removes the dependence on auxiliary 2D or 3D reference motion, unlike reference-motion-driven methods. Its key–value memory aligns text-derived keys with motion values across speakers, while the stylization module rescales the memory slots based on audio-derived style features. In the paper’s interpretation, this arrangement reduces domain gaps and avoids direct regression from audio to personalized motion (Kim et al., 28 Jul 2025).

Subsequent work has clarified the boundaries of this design. The later causal facial motion system Fallingwater characterizes MemoryTalker as a non-streaming and partially personalized baseline. In that comparison, MemoryTalker is described as using an audio-guided style bank and audio-only retrieval, whereas Fallingwater uses dynamic multi-modal retrieval from joint audio and motion context over unstructured variable-length template libraries. The quantitative gap reported in that later study is substantial: MemoryTalker records Sync 0.345, Sim 0.117, ID-FED 67.85, and ID-FPD 6.58, while Fallingwater reports Sync 0.509, Sim 0.718, ID-FED 17.58, and ID-FPD 2.07 under zero-lookahead streaming. In a 15-participant user study with 450 comparisons, Fallingwater is preferred over MemoryTalker by 86.9% on lip sync, 73.8% on identity similarity, and 85.5% on naturalness (Chu et al., 26 Apr 2026).

These later results do not negate MemoryTalker’s original contribution; rather, they help localize it historically. MemoryTalker addressed the problem of audio-only personalization without labels or reference motion. Fallingwater addresses a different frontier: strict streaming causality, dynamic retrieval from unstructured style references, and causal autoregressive decoding. A plausible implication is that MemoryTalker should be read as a strong audio-only personalization architecture within offline or non-streaming settings, rather than as a complete solution to real-time causal facial animation (Chu et al., 26 Apr 2026).

6. Applications, limitations, and research significance

The applications explicitly identified for MemoryTalker include VR telepresence, gaming, film character animation, and virtual avatars in metaverse environments. The combination of audio-only inference, high speed, and personalization makes it suitable for real-time deployment scenarios in which speaker identity labels or reference motions cannot be assumed (Kim et al., 28 Jul 2025).

The framework’s principal strengths are operational rather than purely architectural. It requires only raw audio at inference, does not depend on identity labels, does not need reference 3D motion or 2D videos, generalizes to unseen speakers by separating content and style, and improves temporal stability through the velocity loss. These properties make it practically different from both label-conditioned and reference-conditioned personalization methods (Kim et al., 28 Jul 2025).

The limitations identified by the paper are equally specific. Extreme styles or emotions absent from training may be underrepresented because the stylization mechanism is multiplicative over learned slots. Extreme facial deformations or non-speech motions may therefore not be captured adequately. Audio quality also matters: noise and room acoustics can affect mel-based style features, and very large pronunciation or prosodic deviations may challenge the style encoder. The paper suggests accent normalization, multi-scale prosody modeling, richer slot transformations, optional video cues, and diffusion-based decoders as future directions (Kim et al., 28 Jul 2025).

Within the broader literature, MemoryTalker is significant because it reframed personalized speech-driven 3D facial animation around a memory-based decomposition of content and style. Its central claim is not merely that audio can drive personalized motion, but that audio can do so without identity priors or motion exemplars at inference if the model first learns a reusable bank of general motion patterns and then stylizes those patterns with speaker-specific cues. Later work on causal and retrieval-enhanced animation suggests where that formulation can be extended, but the original system remains an important reference point for audio-only personalized 3D facial animation (Kim et al., 28 Jul 2025).