NVIDIA Audio2Face-3D: Real-Time Facial Animation

• NVIDIA Audio2Face-3D is a real-time, audio-driven system that synthesizes 3D facial motion—including skin, tongue, jaw, and eyeball movements—from speech input.
• It integrates capture-based training, neural audio-to-motion inference, and blendshape retargeting to produce production-ready animations for digital avatars and game characters.
• The platform offers both regression and diffusion model families with open-sourced tools and ARKit compatibility to enhance deployment in virtual assistant and content creation workflows.

NVIDIA Audio2Face-3D is a real-time, audio-driven facial animation system for digital avatars and game characters that converts speech audio, with optional emotion input, into 3D facial motion covering skin, tongue, jaw, and eyeballs. In the formulation described by NVIDIA, the system is designed both for real-time interaction, such as conversational AI avatars and virtual assistants, and for offline content creation, such as generating large amounts of character animation for games. Its technical profile combines capture-based training data, neural audio-to-motion inference, post-processing, and retargeting through blendshape solving, and it is accompanied by open-sourced networks, an SDK, a training framework, and an example dataset (NVIDIA et al., 22 Aug 2025).

1. Historical placement and conceptual scope

Audio2Face-3D belongs to the audio-driven 3D facial animation lineage in which speech is mapped directly to facial control signals rather than to rendered video. An earlier academic precursor, "Audio2Face: Generating Speech/Face Animation from Single Audio with Attention-Based Bidirectional LSTM Networks" (Tian et al., 2019), framed the problem as end-to-end prediction of blendshape parameters from audio only, without images, transcript input, or manual intervention. That earlier system used two bidirectional LSTM layers, an attention mechanism, and dense output layers to predict 51-dimensional blendshape vectors for speech animation and broader facial animation at real-time rates. This suggests a clear continuity of objective: audio-only facial driving, low-latency inference, and retargetable 3D facial control.

Within that trajectory, Audio2Face-3D extends the problem definition beyond lip motion alone. Its target output is not restricted to mouth articulation, and its design explicitly includes tongue, jaw, and eyeball motion, as well as optional emotional conditioning. The system is therefore better understood as a production-oriented facial performance synthesis stack rather than a narrowly scoped lip-sync network.

A second important point of scope is that Audio2Face-3D separates source-motion generation from target-character deployment. Neural models predict dense facial motion tied to source identities, after which a blendshape solver retargets that motion to production rigs. This division between generative inference and rig-compatible retargeting is central to its applicability in games, avatars, and digital content creation workflows.

2. Capture pipeline, data representation, and augmentation

The training corpus described for Audio2Face-3D is built from professional actors recorded with a commercial facial capture system, DI4D, using multiple machine vision cameras, with synchronized audio and facial animation. Each actor performs about 50–70 sentences of length 3–15 seconds in English and Mandarin, and the captured sequences are annotated with 11 emotion states: neutral, amazement, anger, cheekiness, disgust, fear, grief, joy, out-of-breath, pain, and sadness. Abrupt emotion transitions at sequence boundaries are clipped away so that training uses stable emotional segments (NVIDIA et al., 22 Aug 2025).

The system factorizes facial motion into several components, each with its own compact representation. Skin deformation is compressed with PCA; the default skin representation is 140 dimensions, while the Mark model uses 272 dimensions. Tongue motion is handled through a hybrid workflow: a professional animator manually animates one actor’s tongue, a temporary model is trained on that actor, the model generates tongue motion for other actors from audio, and a simple affine transform adapts the result to each target actor. The resulting tongue motion is compressed into a 10-dimensional PCA vector. Jaw motion is represented as rigid motion using five stable vertices—left-back, right-back, left-front, right-front, and center-front—whose per-frame 3D displacements yield a 15-dimensional representation together with neutral positions. Eyeball motion is refined into rig-space yaw and pitch, giving a 4-dimensional eye representation.

These representations impose a structured decomposition on the target space. For the regression model, the resulting output vector is 169-dimensional:

• 140 skin PCA coefficients
• 10 tongue coefficients
• 15 jaw dimensions
• 4 eye dimensions

To expand diversity beyond the captured corpus, Audio2Face-3D applies three augmentation strategies. Voice conversion changes speaker identity while preserving timing and rhythm. Text-to-speech augmentation is combined with phoneme detection and Dynamic Time Warping so that the original animation can be aligned to the new speech signal despite altered rhythm or prosody. Silence augmentation adds 4 seconds of silence for each emotional state to improve behavior on silent or near-silent input. Taken together, these design choices indicate that the system treats articulation, expressivity, and deployment robustness as joint requirements rather than separate subsystems.

3. Model families and learning objectives

Audio2Face-3D comprises two main open-weight model families with distinct operating points in the speed–quality–generality trade space (NVIDIA et al., 22 Aug 2025).

Model family	Core characteristics	Output format
Audio2Face-3D-v2.3	Regression-based; trained separately for Mark, Claire, and James; low-latency and lightweight	1 frame at a time in compressed PCA format
Audio2Face-3D-v3.0	Diffusion-based; supports multiple identities in one model; better expressiveness and quality; streaming-friendly	30 frames at a time in raw vertex format

The regression model takes audio, explicit emotion, and learned implicit emotion. Its audio encoder is hybrid, combining autocorrelation features with Wav2Vec 2.0 features, specifically the 4th transformer layer output with frozen weights. NVIDIA states that this improves lip sync quality, multilingual speech handling, pitch and volume cues, and singing and non-verbal audio. An auxiliary phoneme classifier is trained jointly with cross-entropy loss and later removed after training; its purpose is to shape the encoder through gradients, particularly for bilabial and articulation-related mouth shapes. A convolutional and linear animation decoder then predicts full face pose from the audio features and the emotion vector

$$e = \text{concat}(e^{\text{explicit}}, e^{\text{implicit}}).$$

The regression objective combines multiple terms: MSE loss on facial coefficients, motion loss on frame-to-frame velocity, phoneme loss, phoneme motion loss, volume stability loss for quiet audio, emotion smoothness, lip distance loss, and lip size loss. Identity-specific loss weights are used, with especially heavy weighting of lip-related losses. Inference operates on a sliding window of 8320 samples at 16 kHz with stride $s = 16000/F$, where $F$ is frame rate, enabling streaming frame-by-frame generation.

The diffusion model, Audio2Face-3D-v3.0, is conditioned on HuBERT audio features, noisy motion sequence $x_t$, diffusion timestep $t$, emotion vector, and identity vector. Projected conditioning features are concatenated, processed with GRU layers, and decoded into denoised facial animation. Rather than predicting noise, the model directly predicts denoised motion:

$$\hat{x}_0 = f_\theta(x_t, t, \text{audio}, \text{emotion}, \text{identity}).$$

Its total loss combines a simple MSE denoising term with upper-face smoothness regularization and lip distance loss. For real-time use, it consumes a 1-second audio window, generates the central 0.5 seconds, slides by 0.5 seconds, and reuses the GRU hidden state. NVIDIA reports that 2 diffusion steps are sufficient for good quality at inference, which is the principal reason the diffusion formulation remains practical for streaming.

4. Retargeting, ARKit compatibility, and deployment

A defining feature of Audio2Face-3D is that neural inference does not terminate at source-identity geometry. To deploy on arbitrary characters, the system provides a blendshape solver that converts dense geometry into rig-compatible controls (NVIDIA et al., 22 Aug 2025). Using neutral mesh $x_0$ and blendshape deltas $\Delta_i = x_i - x_0$, expression is modeled as

$x \approx x_0 + \sum_{i=1}^{N} w_i \Delta_i.$

With $\Delta = x - x_0$ and $s = 16000/F$0, the solver estimates weights $s = 16000/F$1 by minimizing

$s = 16000/F$2

The solve is constrained in several production-relevant ways. It uses bounded weights in $s = 16000/F$3, supports an active blendshape subset, canceling poses through a two-pass solve, and symmetric poses via a symmetry regularizer. Vertex fitting is restricted to a salient subset concentrated on the frontal face, with denser sampling around lips, eyelids, and nasolabial folds. A further jaw-driven term is added because a pure surface fit can produce a visually plausible face while yielding physically incorrect jaw motion. This soft jaw term supervises jaw displacement more strongly when the jaw is visually important and when lip separation exposes jaw motion.

ARKit compatibility is a central practical outcome of the solver. The paper states that it can produce ARKit weights for integration with Epic MetaHuman, custom rigs, and other production workflows. This ARKit-facing design is also the aspect most often emphasized in third-party deployment literature, where Audio2Face is treated as a production-ready benchmark for Unreal Engine pipelines and ARKit-style output.

The deployment story is reinforced by the open-sourced release. NVIDIA explicitly states that it has open-sourced the regression and diffusion model weights, the SDK, the training framework, and an example dataset, Audio2Face-3D-Dataset-v1.0.0-claire. It also releases pretrained Audio2Emotion ONNX models, A2E-v2.2 and A2E-v3.0, although the standalone Audio2Emotion model is not released as a separate standalone product under the same license constraints.

5. Evaluation methodology, runtime, and external benchmarking

Audio2Face-3D uses four internal benchmark metrics for quality assessment, although NVIDIA states that expert subjective inspection remains the main evaluation criterion (NVIDIA et al., 22 Aug 2025). SyncNet score, based on a pretrained 3D SyncNet from GeneFace, measures audio–mouth correspondence on a $s = 16000/F$4 scale with higher being better. The jitter metric measures high-frequency instability in the mouth region by combining Fourier or high-frequency intensity with Fréchet distance between ground-truth and predicted vertex animations. The bilabial sound score evaluates closure quality for /M/, /B/, and /P/ by aligning audio to phonemes, extracting bilabial intervals, and checking whether minimum upper–lower lip distance falls below a threshold. The expressiveness score measures emotional intensity using neutralized FACEM features. NVIDIA characterizes these metrics primarily as sanity checks.

The reported inference performance is high. On an RTX 4090, v2.3-Claire reaches 194 FPS in PyTorch, 252 FPS in ONNX, and 453 FPS in TensorRT. The v3.0 model reaches 2069 FPS in PyTorch, 1803 FPS in ONNX, and 3269 FPS in TensorRT. Memory usage is reported as about 0.6 GB for v2.3 TensorRT with 8 tracks, and about 1.4 GB for v3.0 TensorRT with 1 track and 4.0 GB with 8 tracks. NVIDIA notes that v3.0 uses more memory because it outputs 30 frames in raw vertex format rather than one compressed frame.

An independent Unreal Engine deployment study provides an external production comparison. In that work, NVIDIA Audio2Face is treated as a commercial baseline that already integrates with Unreal Engine and produces ARKit-compatible facial animation; the authors state that it is trained on subject-specific 4D scans adapted to solve for ARKit blendshapes. In two perceptual studies comparing Epic Games’ MetaHuman speech-driven animator, NVIDIA Audio2Face, and two retrained academic baselines, the consistent ranking was EG > NV > FD/PT, with NVIDIA Audio2Face second-best in Lip-Sync and Realism across both studies. In Experiment 1, emotion recognition accuracy was 71.11% for EG, 55.00% for NV, 51.11% for FD, and 49.72% for PT (Busacchi et al., 9 Jun 2026). These findings place Audio2Face-3D in a strong production-quality position, albeit below Epic’s system in that particular benchmark.

6. Relation to adjacent research, common misconceptions, and limitations

Audio2Face-3D occupies a specific position among neighboring audio-driven avatar systems. EmoFace, for example, is a model-based, emotion-controllable audio-driven 3D face animation system intended to drive MetaHuman controller rigs rather than pixels or meshes. It uses wav2vec2.0 BASE, a separate emotion encoder, a Transformer-encoder-based Audio2Rig module, and outputs a 174-dimensional controller sequence per frame. It also adds explicit blink and gaze post-processing, including probabilistic blink sampling and randomized gaze, and is positioned for game NPC dialogue and VR avatars (Liu et al., 2024). Relative to Audio2Face-3D, this defines a more specialized rig-control regime with explicit discrete emotion labels and a MetaHuman-centric controller space.

VASA-3D, by contrast, represents a different branch of the field. It takes a learned VASA-1 motion latent, maps it into FLAME parameters with MLPs, adds latent-conditioned residual Gaussian deformations, and renders free-viewpoint 3D talking heads by Gaussian splatting. Its stated distinction from many Audio2Face-3D-like systems is that control is not confined to a conventional low-dimensional mesh expression space, but instead uses a learned latent that captures richer expression and pose variation than standard parametric facial rigs (Xu et al., 16 Dec 2025). This suggests a divergence between two design philosophies: retargetable production rigging on the one hand, and latent-driven avatar expressivity with single-image customization on the other.

Several misconceptions recur around Audio2Face-3D. One is that it is merely a lip-sync system. The technical description contradicts this directly: the system predicts motion for skin, tongue, jaw, and eyeballs, supports explicit and implicit emotion conditioning, and includes an auxiliary Audio2Emotion pathway. Another is that it should be understood as a single architecture. In fact, the platform includes both regression-based and diffusion-based families, with different identity assumptions, latency profiles, and output representations. A third concerns openness. NVIDIA states that it has open-sourced networks, SDK, training framework, and an example dataset, while third-party deployment work groups Audio2Face with closed commercial tools in the sense that external models cannot be integrated into those production systems. The two descriptions are not strictly incompatible: one concerns released technical assets, while the other concerns the extensibility of the packaged commercial workflow.

The limitations reported for Audio2Face-3D are also precise. Performance degrades with severe background noise; the system struggles with strong non-verbal audio; it may fail on lip shapes not seen in training; upper-face and eyeball motion are not semantically rich enough; it does not naturally generate convincing idle or listening behavior from long silence or partner speech; and short audio windows limit broader conversational and contextual understanding (NVIDIA et al., 22 Aug 2025). These constraints indicate that the system is strongest at speech-driven local facial articulation and weaker at context-aware social behavior. In that sense, current Audio2Face-3D remains a high-performance speech-to-face animation framework rather than a full conversational behavior model.