---
title: 'MemoryTalker: Audio-Driven 3D Facial Animation'
url: https://www.emergentmind.com/topics/memorytalker
type: topic
---

# MemoryTalker: Audio-Driven 3D Facial Animation

MemoryTalker is a speech-driven 3D facial animation framework that synthesizes personalized facial motion sequences from audio alone, without requiring identity labels, reference 2D videos, or reference 3D meshes at inference time. It was introduced to address a central limitation of prior personalization strategies: one-hot identity conditioning does not generalize to unseen speakers, while reference-motion-driven methods depend on auxiliary motion inputs that are impractical in real-time deployment. The framework is organized as a two-stage pipeline—“Memorizing” and “Animating”—in which a multimodal motion memory stores general facial motion patterns, and an audio-guided stylization module modulates that memory according to speaker-specific cues extracted from the input audio [2507.20562].

## 1. Problem formulation and design motivation

Speech-driven 3D facial animation aims to synthesize temporally coherent 3D face mesh motions synchronized with input speech. In the formulation used by MemoryTalker, high-quality synthesis is not limited to lip synchronization; it must also capture personal speaking style, including mouth opening amplitude, pouting extent, and subtle articulation differences across speakers [2507.20562].

The framework is motivated by two deficiencies in earlier personalization paradigms. First, one-hot identity conditioning, used by systems such as VOCA, FaceFormer, and CodeTalker, encodes speaker identity during training and requires selecting an identity at inference time. This setup fails for unseen speakers and cannot recover their style without labels. Second, reference-motion-driven approaches such as Imitator, Composite/Regional Movements, Mimic, and Probabilistic SD 3D Facial can adapt style from auxiliary 2D video or 3D mesh sequences, but they require additional inputs that are often unavailable in real-time applications [2507.20562].

MemoryTalker therefore targets a narrower but operationally important objective: arbitrary-speaker personalization from raw audio only. Its central design choice is to decouple content and style. Content is represented by ASR-derived, speaker-agnostic text features, while style is derived from mel-spectrogram features that encode prosody, rhythm, and timbre. This separation allows the system to retrieve general motion patterns from memory and then emphasize the motion types most compatible with the current speaker’s speaking style [2507.20562].

A common misunderstanding is to read the word “memory” as implying identity lookup or explicit user-specific memory storage. In MemoryTalker, the memory is instead a learnable bank of general facial motion slots. Personalization is achieved by modulating those slots with audio-driven style weights, not by retrieving a stored subject-specific template at inference time [2507.20562].

## 2. Representation and architectural components

The model represents facial motion as per-vertex displacements relative to a neutral registered mesh template. The facial motion at time $t$ is denoted $v^t \in \mathbb{R}^3$, corresponding to per-vertex 3D movement with respect to the neutral face. On VOCASET, the mesh topology is FLAME with 5,023 vertices at 60 fps; on BIWI, the topology has 23,370 vertices at 25 fps, with average sequence length approximately 4.67 s [2507.20562].

The architecture has five principal components. The audio-to-text encoder $E_{aud}$ is a pretrained HuBERT ASR encoder. After interpolation to match the video frame rate, a projection layer maps the output to memory-key channels:
$$
f_{txt}^{1:T} = \psi_{\to n}(\mathrm{Interp}(E_{aud}(a))) \in \mathbb{R}^{T \times n}.
$$
These features are intended to encode phonetic content in a speaker-agnostic form [2507.20562].

The motion encoder $E_m$ is a single linear layer that maps per-frame vertex displacements into a latent motion feature $f_m^t \in \mathbb{R}^c$. The learnable motion memory $M_m$ contains $n$ slots, each a $c$-dimensional vector:
$$
M_m = \{s_m^i\}_{i=1}^n \in \mathbb{R}^{n \times c}.
$$
During memorization, the model computes value-addressing weights from the motion feature using cosine similarity:
$$
w_m^i = \frac{\exp(\kappa \cdot d(s_m^i, f_m^t))}{\sum_{j=1}^n \exp(\kappa \cdot d(s_m^j, f_m^t))},
$$
and reconstructs a recalled motion feature
$$
\hat f_{m,val}^t = \sum_{i=1}^n w_m^i \cdot s_m^i.
$$
A parallel key-addressing path uses the ASR-derived text representation:
$$
K_{txt}^t = \mathrm{softmax}(f_{txt}^t) \in \mathbb{R}^n,
$$
with general motion recall
$$
\hat f_{m,key}^t = \sum_{i=1}^n w_{txt}^i \cdot s_m^i.
$$
The paper presents this memory as a bridge between audio-text keys and motion values across speakers [2507.20562].

Personalization is introduced by a speaking style encoder $E_s$ that processes mel-spectrograms $\phi^{a2m}(a)$ with 80 mel bins. According to the supplementary implementation summary, this module uses Conv1d stacks, GroupNorm, pooling, and MLP layers to produce a style feature $f_s$. The style feature is converted into per-slot weights,
$$
\tilde w_s = \{\tilde w_s^i\}_{i=1}^n = \mathrm{sigmoid}(\psi'_{\to n}(f_s)) \cdot \psi_{\to 1}(f_s),
$$
which scale the motion memory slotwise:
$$
\tilde M_m = \{\tilde s_m^i\}_{i=1}^n = \{\tilde w_s^i \cdot s_m^i\}_{i=1}^n.
$$
Stylized recall then becomes
$$
\tilde f_{m,key}^t = \sum_{i=1}^n w_{txt}^i \cdot \tilde s_m^i.
$$
This multiplicative modulation is the mechanism by which MemoryTalker emphasizes motion types associated with a particular speaking style [2507.20562].

The temporal decoder $D_m$ is transformer-based, following the FaceFormer style of sequence decoding. It reconstructs either general motion,
$$
\hat v^t = D_m([f_{txt}^t; \hat f_{m,key}^t], f_{txt}^t),
$$
or personalized motion,
$$
\tilde v^t = D_m([f_{txt}^t; \tilde f_{m,key}^t], f_{txt}^t).
$$
The overall architecture is therefore neither a direct audio-to-mesh regressor nor a template transfer system; it is a memory-conditioned decoder in which content selects motion slots and style rescales them [2507.20562].

## 3. Two-stage training procedure

MemoryTalker is trained in two stages. The first stage, “Memorizing,” learns to store and retrieve general motion patterns independently of speaker style. The second stage, “Animating,” freezes the first-stage modules and trains only the style encoder so that audio-derived style information can modulate the memory for personalized synthesis [2507.20562].

In the memorizing stage, the model aligns motion-derived value addresses with text-derived key addresses. The alignment loss is
$$
\mathcal{L}_{align} = \sum_{t=1}^T K_{txt}^t \cdot \log\left(\frac{K_{txt}^t}{V_m^t}\right),
$$
and the memory reconstruction loss is
$$
\mathcal{L}_{mem} = \sum_{t=1}^T \| f_m^t - \hat f_{m,val}^t \|_2^2.
$$
Sequence reconstruction is supervised with mesh reconstruction loss
$$
\mathcal{L}_{mse} = \sum_{t=1}^T \| v^t - \hat v^t \|_2^2
$$
and velocity consistency loss
$$
\mathcal{L}_{vel} = \sum_{t=0}^{T-1} \| (v^{t+1} - v^t) - (\hat v^{t+1} - \hat v^t) \|_2^2.
$$
The total first-stage objective is
$$
\mathcal{L}_{1\text{-stage}} = \mathcal{L}_{mse} + \mathcal{L}_{vel} + \lambda_1(\mathcal{L}_{mem} + \mathcal{L}_{align}),
$$
with $\lambda_1 = 0.01$ [2507.20562].

In the animating stage, the style encoder receives mel-spectrograms and is trained with a triplet-style loss,
$$
\mathcal{L}_{style} = \max\left(\|f_s - f_s^p\|_2^2 - \|f_s - f_s^n\|_2^2 + l, 0\right),
$$
where positive and negative style features are drawn from same-speaker and different-speaker samples, respectively. The model also uses a lip-region emphasis term,
$$
\mathcal{L}_{lip} = \text{mean over lip-region vertices of } \| v^t - \tilde v^t \|_2^2.
$$
The second-stage objective is
$$
\mathcal{L}_{2\text{-stage}} = \mathcal{L}_{mse} + \mathcal{L}_{vel} + \lambda_2(\mathcal{L}_{lip} + \mathcal{L}_{style}),
$$
with $\lambda_2 = 0.01$ [2507.20562].

The training protocol follows FaceFormer and CodeTalker splits. On VOCASET, the data comprise 480 sequences, 12 subjects, and 255 sentences at 60 fps, with approximately 4 s per sequence. BIWI contains 40 shared sentences across 14 subjects, including neutral and emotional variants, at 25 fps. Stage 1 trains all modules except $E_s$ for 100 epochs with learning rate $10^{-4}$ on an NVIDIA A6000 GPU. Stage 2 freezes all stage-1 modules and trains only $E_s$ for 100 epochs with learning rate $5 \times 10^{-5}$. The number of memory slots is fixed to $n = 32$, selected by ablation among $\{16,24,32,48,64\}$ [2507.20562].

This two-stage decomposition suggests a deliberate bias toward content stability before style specialization. The paper’s end-to-end comparison supports that reading: on VOCASET, end-to-end training yields FVE 0.510, LVE 0.303, and FID 3.142, whereas the two-stage setup yields 0.506, 0.293, and 3.045 [2507.20562].

## 4. Evaluation protocol and empirical performance

The evaluation uses Face Vertex Error (FVE), Lip Vertex Error (LVE), Lip Dynamic Time Warping (LDTW), Fréchet Inception Distance (FID), and Lip-max. FVE is the mean Euclidean error across face vertices and frames; LVE restricts that computation to lower-face or lip vertices; LDTW measures temporal similarity of lip trajectories; FID is computed on rendered images; and Lip-max captures the average of the maximum per-frame lip-region vertex error [2507.20562].

On VOCASET, MemoryTalker reports FVE $0.506 \times 10^{-6}$, LVE $0.293 \times 10^{-5}$, FID $3.045 \times 10^{-1}$, LDTW $0.418 \times 10^{-5}$, and Lip-max $0.331 \times 10^{-4}$. The paper states that these results surpass FaceFormer, which records $(0.639, 0.413, 3.583, 0.507, 0.452)$, and UniTalker, which records $(0.570, 0.382, 3.256, 0.507, 0.407)$ in the same metric order. On BIWI, MemoryTalker reports FVE $0.901 \times 10^{-4}$, LVE $0.187 \times 10^{-4}$, FID $7.202 \times 10^{-1}$, LDTW $0.107 \times 10^{-4}$, and Lip-max $0.398 \times 10^{-3}$, surpassing FaceFormer and CodeTalker according to the reported comparison [2507.20562].

The paper also isolates an instability in one-hot identity-conditioned models. For the same audio input on VOCASET, FaceFormer’s FVE varies as $0.639 \pm 0.036$, CodeTalker’s as $0.721 \pm 0.056$, and Imitator’s as $0.686 \pm 0.069$, depending on the selected training identity. MemoryTalker produces a single result, reported as FVE 0.506, LVE 0.293, and FID 3.045, because it does not require identity labels at inference [2507.20562].

Ablation results attribute performance gains to both the memory stage and the style stage. On VOCASET, a baseline without memory or style obtains FVE 0.638 and LVE 0.460; adding first-stage memory alone improves these to 0.531 and 0.313; the full two-stage model reaches 0.506 and 0.293. Removing $\mathcal{L}_{lip}$ or $\mathcal{L}_{style}$ yields modest degradations, for example FVE 0.514 or 0.513 instead of 0.506, and LDTW 0.435 or 0.431 instead of 0.418. A separate comparison between mel-based and ASR-derived style features finds that mel-based $E_s$ performs better on VOCASET, with FVE 0.506 versus 0.518, LVE 0.293 versus 0.300, and LDTW 0.418 versus 0.437 [2507.20562].

The runtime profile is one of the framework’s notable practical attributes. For 1 s of audio, MemoryTalker requires 7.8 ms of inference time on an NVIDIA A6000 GPU, corresponding to approximately 120–128 fps, with 94M parameters. In the reported efficiency comparison, this is faster than FaceFormer at 38.1 ms, CodeTalker at 297.6 ms, SelfTalk at 10.1 ms, and UniTalker at 9.7 ms, while remaining substantially smaller than CodeTalker and UniTalker in parameter count [2507.20562].

Subjective evaluation is reported on VOCASET with 33 participants in A/B tests. MemoryTalker is preferred over FaceFormer, CodeTalker, Imitator, ScanTalk, and UniTalker across lip-sync, realism, and speaking style. For example, against FaceFormer it receives 83.9% preference in lip-sync, 85.5% in realism, and 80.6% in style; against ScanTalk the corresponding figures are 94.4%, 91.1%, and 92.8%. Against ground truth, MemoryTalker is selected 41.1% of the time for lip sync and 40.3% for realism, which the paper presents as indicating proximity to the ground-truth sequences in subjective assessment [2507.20562].

## 5. Relation to prior and subsequent systems

Relative to earlier personalized facial animation methods, MemoryTalker occupies a distinct position. It removes identity priors at inference, unlike one-hot-conditioned methods, and removes the dependence on auxiliary 2D or 3D reference motion, unlike reference-motion-driven methods. Its key–value memory aligns text-derived keys with motion values across speakers, while the stylization module rescales the memory slots based on audio-derived style features. In the paper’s interpretation, this arrangement reduces domain gaps and avoids direct regression from audio to personalized motion [2507.20562].

Subsequent work has clarified the boundaries of this design. The later causal facial motion system Fallingwater characterizes MemoryTalker as a non-streaming and partially personalized baseline. In that comparison, MemoryTalker is described as using an audio-guided style bank and audio-only retrieval, whereas Fallingwater uses dynamic multi-modal retrieval from joint audio and motion context over unstructured variable-length template libraries. The quantitative gap reported in that later study is substantial: MemoryTalker records Sync 0.345, Sim 0.117, ID-FED 67.85, and ID-FPD 6.58, while Fallingwater reports Sync 0.509, Sim 0.718, ID-FED 17.58, and ID-FPD 2.07 under zero-lookahead streaming. In a 15-participant user study with 450 comparisons, Fallingwater is preferred over MemoryTalker by 86.9% on lip sync, 73.8% on identity similarity, and 85.5% on naturalness [2604.23692].

These later results do not negate MemoryTalker’s original contribution; rather, they help localize it historically. MemoryTalker addressed the problem of audio-only personalization without labels or reference motion. Fallingwater addresses a different frontier: strict streaming causality, dynamic retrieval from unstructured style references, and causal autoregressive decoding. A plausible implication is that MemoryTalker should be read as a strong audio-only personalization architecture within offline or non-streaming settings, rather than as a complete solution to real-time causal facial animation [2604.23692].

## 6. Applications, limitations, and research significance

The applications explicitly identified for MemoryTalker include VR telepresence, gaming, film character animation, and virtual avatars in metaverse environments. The combination of audio-only inference, high speed, and personalization makes it suitable for real-time deployment scenarios in which speaker identity labels or reference motions cannot be assumed [2507.20562].

The framework’s principal strengths are operational rather than purely architectural. It requires only raw audio at inference, does not depend on identity labels, does not need reference 3D motion or 2D videos, generalizes to unseen speakers by separating content and style, and improves temporal stability through the velocity loss. These properties make it practically different from both label-conditioned and reference-conditioned personalization methods [2507.20562].

The limitations identified by the paper are equally specific. Extreme styles or emotions absent from training may be underrepresented because the stylization mechanism is multiplicative over learned slots. Extreme facial deformations or non-speech motions may therefore not be captured adequately. Audio quality also matters: noise and room acoustics can affect mel-based style features, and very large pronunciation or prosodic deviations may challenge the style encoder. The paper suggests accent normalization, multi-scale prosody modeling, richer slot transformations, optional video cues, and diffusion-based decoders as future directions [2507.20562].

Within the broader literature, MemoryTalker is significant because it reframed personalized speech-driven 3D facial animation around a memory-based decomposition of content and style. Its central claim is not merely that audio can drive personalized motion, but that audio can do so without identity priors or motion exemplars at inference if the model first learns a reusable bank of general motion patterns and then stylizes those patterns with speaker-specific cues. Later work on causal and retrieval-enhanced animation suggests where that formulation can be extended, but the original system remains an important reference point for audio-only personalized 3D facial animation [2507.20562].

Source: https://www.emergentmind.com/topics/memorytalker