VisemeNet: Deep Learning for Speech Animation

• VisemeNet is a deep learning framework for speech animation that generates editable viseme motion curves aligned with production-quality facial rigs.
• It employs a three-stage LSTM architecture using feature extraction, phoneme grouping, and landmark prediction to ensure precise, real-time lip synchronization.
• The system achieves high precision with low error rates, offering animators robust control and seamless integration into existing production pipelines.

VisemeNet is a deep learning framework for audio-driven, animator-centric speech animation, designed to generate highly editable viseme motion curves compatible with industry-standard facial animation rigs such as JALI and FACS. The system processes raw input audio and outputs time-indexed curves for 29 rig parameters, thereby delivering real-time, high-fidelity lip synchronization suitable for production pipelines and animator workflows (Zhou et al., 2018).

1. System Structure and Processing Pipeline

VisemeNet employs a three-stage Long Short-Term Memory (LSTM) network architecture. The input audio is subjected to feature extraction at 100 frames per second, resulting in per-frame feature vectors $\mathbf{x}_t \in \mathbb{R}^{65}$ composed of 13 MFCCs, 26 Mel filter-bank energies, and 26 spectral subband centroids. For each frame at time $t$, a $\pm11$ frame context window is used, yielding $1560$-dimensional inputs reflecting approximately $240$ ms of audio, corresponding to the average phoneme duration.

The architecture is divided into three interconnected stages:

1. Phoneme-Group Stage: Maps input features to 20 visually distinct phonetic groups, performing classification via three stacked unidirectional LSTM layers (256 hidden units each) and a softmax decoder.
2. Landmark Stage: Predicts 38 lower-face 2D landmark displacements (jaw, lips, nose) relative to neutral, leveraging features of the first stage through multi-task sharing.
3. Viseme Stage: Fuses outputs from the first two stages with original input features and propagates them through three parallel LSTM-branch decoders to predict:
   • 29 continuous JALI rig parameters (20 viseme intensities, 9 co-articulation controls),
   • 29 binary activation flags,
   • 2D viseme-field style features.

Final outputs are sparse, editable curves suitable for real-time (~1 ms compute per frame; total lag ~121 ms) integration with production rigging systems (Zhou et al., 2018).

2. Psycho-Linguistic Motivation

The design of VisemeNet is informed by several psycho-linguistic insights:

• Phonetic-Group Collapsing for Viseme Generation: Given that multiple International Phonetic Alphabet (IPA) phonemes (such as /b/, /p/) are visually confusable and map to the same viseme, the system reduces the phoneme space to 20 viseme-relevant groups. This simplification converts the recognition task from a complex phoneme-to-viseme mapping to a 20-way classification, aligning with perceptual studies on speech recognition.
• Speech Style Cues from Landmark Motion: Variability in speaker delivery (e.g., mumbling, shouting, whispering) influences the trajectory of facial landmarks. By explicitly predicting landmark displacements, the model captures stylistic aspects tied to prosody and emotion.
• Animator-Centric Motion-Curve Representation: Industry keyframes encode nuanced animator intent in terms of onset, apex, sustain, and decay of viseme activation. VisemeNet learns to emit directly editable continuous curves, incorporating these temporal dynamics and facilitating keyframe-style editing.

3. Audio Feature Extraction

Audio preprocessing is performed at 100 Hz using a 25 ms Hamming window. Each frame's feature vector comprises 13 MFCCs, 26 filter-bank energies, and 26 spectral subband centroids, for a total of 65 dimensions per frame. The network operates over input windows of 24 frames (±11 around $t$), resulting in 1560-dimensional context vectors. This design enables the capture of temporal context necessary for accurate viseme activation and transition detection.

4. Multi-Stage LSTM Architecture and Motion-Curve Output

The LSTM cell update equations for each stage at time $t$ are as follows:

$$\begin{aligned} i_t &= \sigma(W_i x_t + U_i h_{t-1} + b_i) \ f_t &= \sigma(W_f x_t + U_f h_{t-1} + b_f) \ o_t &= \sigma(W_o x_t + U_o h_{t-1} + b_o) \ \tilde{g}_t &= \tanh(W_g x_t + U_g h_{t-1} + b_g) \ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{g}_t \ h_t &= o_t \odot \tanh(c_t) \end{aligned}$$

Decoder and Output Pathways:

• Phoneme-Group Output: Softmax over 20 phonetic groups per frame.
• Landmark Output: Predicts 76 values (x, y offsets for 38 landmarks).
• Viseme Output: Three parallel branches (no shared weights):
  • Activation-flags branch: 29D sigmoid outputs for binary activation flags.
  • Continuous-rig branch: 29D outputs for rig parameter values.
  • Viseme-field branch: 2D continuous output encoding broad style/mouth-shape cues.

Inference uses per-control thresholds, determined on a hold-out set, to binarize activation flags and thereby gate the corresponding continuous outputs.

5. Training Objectives and Optimization

VisemeNet employs a two-phase training protocol:

• Pre-training (Phoneme + Landmark Stages):
  • Weighted composite loss: $$L_1(\Theta_{LSTM}, \Theta_{dec(c)}, \Theta_{dec(q)}) = w_c L_c + w_q L_q + w_q' L_q'$$
  • $L_c$: Phoneme-group cross-entropy,
  • $t$0: Landmark $t$1-regression,
  • $t$2: Landmark temporal smoothness.
• Joint Training (All Three Stages):
  • Full loss: $t$3
  • $t$4: Activation-flag cross-entropy,
  • $t$5: Rig-value regression (active frames only),
  • $t$6, $t$7, $t$8: Smoothness and regression for rig and viseme-field values.

Hyperparameter selection for task losses employs small hold-out sets. Optimization uses SGD with momentum, batch size 256, and learning rate $t$9 (Zhou et al., 2018).

6. Integration with Production Pipelines

VisemeNet's outputs are directly compatible with FACS- and JALI-based production rigs. The curves use on/off gating with per-frame intensity, closely matching standard animation industry practice. Real-time inference at 100 FPS with $\pm11$0 ms compute time and $\pm11$1 ms look-ahead ensures negligible latency. The output curves are directly editable, supporting instant animator overrides, keyframe insertion, and other downstream workflows typical in production environments.

7. Experimental Evaluation and Results

Cross-Validation and Dataset

Experiments use a one-hour, JALI-annotated BIWI subset (14 speakers, neutral and expressive styles) with leave-one-out cross-validation—13 speakers for training and pre-training on GRID/SAVEE, 1 held out for testing.

Metrics and Outcomes

VisemeNet achieves the following average metrics across splits for both neutral and expressive speech:

Metric	Neutral	Expressive
Precision (%)	89.5	90.1
Recall (%)	92.2	92.3
Curve Error (%)	7.8	7.6

Evaluation is based on activation precision/recall and normalized $\pm11$2 motion-curve error over active frames. Variants omitting a stage (e.g., audio-only, no phoneme grouping, no landmark prediction) show 2–7% drop in precision/recall and 5–7% increase in curve error, confirming the necessity of all architectural elements.

Animator Feedback and Comparative Analysis

Professional animator critique confirms that VisemeNet’s JALI curves are sparse, accurately capture keyframe-style motion, and are straightforward to edit. Visual comparisons show that while lip-synchronization accuracy is on par with other deep-learning approaches, VisemeNet delivers superior animator-control and editability for production.

Speaker and Language Robustness

Pre-training on GRID/SAVEE and BIWI enables robust language-agnostic phoneme grouping. Leave-one-out validation verifies consistent performance across both gender and speech style, including neutral and expressive speech (Zhou et al., 2018).

• VisemeNet: Audio-Driven Animator-Centric Speech Animation (Zhou et al., 2018)