AUBlendNet: AU-Controllable 3D Face Rig Generation

• AUBlendNet is a neural framework that generates identity-specific AU-blendshape rigs from a neutral FLAME mesh, enabling continuous 3D facial expression control through 32 standard AUs.
• It employs a two-stage design with an AUCodebook for latent dictionary learning and StyleBlendNet for mapping identity geometry, ensuring efficient, real-time rig synthesis.
• The method leverages the curated AUBlendSet for high-fidelity, stylized expression manipulation and has applications in animation, speech-driven facial synthesis, and AU recognition augmentation.

AUBlendNet is a neural framework for generating stylized, AU-controllable 3D facial expression rigs from a neutral identity mesh. Introduced together with AUBlendSet, it predicts, in parallel, the AU-Blendshape basis vectors of the corresponding style for a given identity mesh, enabling continuous 3D facial expression manipulation across identities through 32 standard Facial Action Units (AUs) (Li et al., 16 Jul 2025). The method is formulated for FLAME-topology faces and is designed to preserve character-specific deformation style while maintaining semantic alignment with FACS-defined AU controls.

1. Problem setting and conceptual scope

AUBlendNet addresses fine-grained, stylized 3D facial expression manipulation across identities. The central objective is to take a 3D face identity, represented as a template mesh, and automatically generate a set of AU-Blendshape basis vectors that are semantically aligned with 32 standard Facial Action Units, stylized to match that specific identity, and suitable for continuous user control (Li et al., 16 Jul 2025).

The problem is motivated by three limitations in prior practice. First, universal control bases such as ARKit or generic FACS-based rigs provide good controllability but limited personalization, because all characters share the same expression basis. Second, commonly used 2D and 3D expression datasets are not designed for controllable 3D expression manipulation: AU annotations are often multi-AU coupled, blendshape bases are absent, or identity-specific stylized rigs are unavailable. Third, the mapping from FACS AU codes to mesh deformation is identity-dependent; the same AU can manifest differently across facial geometries, ages, or stylization regimes.

Within this formulation, AUBlendNet functions as an auto-rigging system for per-identity AU-aligned blendshape spaces. It decouples the generation of a controllable expression basis from the subsequent application of AU coefficients. This suggests a separation between rig synthesis and expression synthesis: the network predicts the basis once, and downstream expression editing reduces to linear combination of those predicted bases.

2. AU-Blendshape representation and AUBlendSet

The underlying representation is a linear blendshape rig in which each basis vector corresponds to one Facial Action Unit. For a neutral mesh $\mathbf{V}_0 \in \mathbb{R}^{V \times 3}$ and $N = 32$ AUs, expression synthesis is written as

$$\mathbf{V}(\mathbf{a}) = \mathbf{V}_0 + \sum_{k=1}^{N} a_k \mathbf{B}_k,$$

where $\mathbf{B}_k \in \mathbb{R}^{V \times 3}$ is the AU-Blendshape basis for AU $k$ and $a_k$ is a continuous AU intensity (Li et al., 16 Jul 2025). In the matrix notation used in the work, the AU-Blendshape bases for one identity are denoted $\mathbf{B} \in \mathbb{R}^{N \times V}$, with the factor of 3 for XYZ absorbed into $V$, and $V = 5023$ for FLAME meshes.

The AU inventory is derived from standard FACS. To balance coverage and redundancy, the authors remove redundant eye AUs that differ mostly by intensity, tongue AUs, and head posture AUs, retaining 32 AUs as control bases. Each retained AU has a semantic definition from FACS and a corresponding per-identity blendshape basis manually sculpted to match canonical AU motion while respecting the facial style of the identity.

AUBlendSet is the dataset that supplies supervision for this representation. It is based on the expressive FLAME model, with each mesh containing 5,023 vertices. The dataset samples the FLAME parameter space over gender, age from 20 to 60, and facial shape proportions, producing 500 identity themes: 206 females and 294 males. For each identity, professional artists create 32 AU-Blendshape control bases guided by FACS AU exemplars and standard animation pipelines for blendshape creation. The bases are manually reviewed for clarity of motion, structural correctness, and consistency with AU semantics. AUBlendSet also provides AU-annotated facial postures per identity, generated by activating one or more AU-Blendshapes with known weights.

A concise comparison given in the paper situates AUBlendSet among related resources.

Dataset	Identities	Blendshapes / AU annotations / editing suitability
FACSD3D	10	AU annotations only; discrete; no blendshapes; low editing suitability
CK+	97	AU annotations only; discrete; no blendshapes; low editing suitability
BP4D	23	AU annotations only; discrete; no blendshapes; low editing suitability
FaceWarehouse	150	Blendshapes available; no AU annotation; entangled bases; low editing suitability
AUBlendSet	500	Blendshapes: yes; AU annotations: yes; continuous control; high editing suitability

The work characterizes AUBlendSet as the first dataset with per-identity stylized AU-Blendshape control bases aligned with 32 AUs, plus AU-annotated expressions (Li et al., 16 Jul 2025). A plausible implication is that the dataset is not merely a corpus of expressions but a rig-level supervision source, which is essential for decoupled and continuous AU-based control.

3. Network architecture and learning formulation

AUBlendNet is organized as a two-stage system: AUCodebook and StyleBlendNet. AUCodebook learns a discrete latent dictionary over AU-Blendshape bases, while StyleBlendNet maps a neutral template mesh into that latent space and reconstructs the corresponding AU-Blendshape rig (Li et al., 16 Jul 2025).

AUCodebook is described as a VAE-like Transformer with vector quantization. Its input is the AU-Blendshape matrix $\mathbf{B} \in \mathbb{R}^{N \times V}$ for one identity. The encoder $N = 32$0 produces latent tokens

$N = 32$1

with latent dimension $N = 32$2. These tokens are quantized to nearest entries in a learned codebook, producing $N = 32$3, and the decoder $N = 32$4 reconstructs the bases:

$N = 32$5

The AUCodebook loss follows a VQ-VAE-style formulation:

$N = 32$6

where $N = 32$7 denotes stop-gradient and $N = 32$8 in experiments. The first term is an $N = 32$9 reconstruction loss over all vertices of all AU bases, and the latter two terms are the standard commitment and codebook update losses.

StyleBlendNet performs the inverse mapping from identity geometry to AUCodebook latent space. Given a template mesh $$\mathbf{V}(\mathbf{a}) = \mathbf{V}_0 + \sum_{k=1}^{N} a_k \mathbf{B}_k,$$0, represented in flattened form and expanded into a sequence of length $$\mathbf{V}(\mathbf{a}) = \mathbf{V}_0 + \sum_{k=1}^{N} a_k \mathbf{B}_k,$$1, the model first applies a Temporal Convolution Network,

$$\mathbf{V}(\mathbf{a}) = \mathbf{V}_0 + \sum_{k=1}^{N} a_k \mathbf{B}_k,$$2

followed by a feature projection MLP,

$$\mathbf{V}(\mathbf{a}) = \mathbf{V}_0 + \sum_{k=1}^{N} a_k \mathbf{B}_k,$$3

Conditioning on facial style is implemented through AdaLN-Zero parameters derived from the template mesh:

$$\mathbf{V}(\mathbf{a}) = \mathbf{V}_0 + \sum_{k=1}^{N} a_k \mathbf{B}_k,$$4

Within each Transformer block, these parameters modulate LayerNorm and residual scaling for the attention and MLP sublayers. The resulting latent tokens are quantized through the AUCodebook dictionary and decoded into predicted AU-Blendshape bases $$\mathbf{V}(\mathbf{a}) = \mathbf{V}_0 + \sum_{k=1}^{N} a_k \mathbf{B}_k,$$5.

The StyleBlendNet training loss is

$$\mathbf{V}(\mathbf{a}) = \mathbf{V}_0 + \sum_{k=1}^{N} a_k \mathbf{B}_k,$$6

A defining property of the method is parallel prediction of the full AU rig. The network outputs all $$\mathbf{V}(\mathbf{a}) = \mathbf{V}_0 + \sum_{k=1}^{N} a_k \mathbf{B}_k,$$7 latent tokens in one forward pass rather than generating bases autoregressively. The paper notes that autoregressive prediction can yield slightly better accuracy but is slower; the chosen non-autoregressive design emphasizes efficiency for real-time facial animation.

4. Training procedure, inference, and implementation

The data representation throughout is a fixed-topology FLAME mesh with 5,023 vertices. For AUBlendSet experiments, each identity provides a neutral mesh, 32 AU-Blendshape bases, and AU-annotated expression examples. For comparison experiments on CK+ and D3DFACS, the work uses 8:1:1 splits, and CK+ images together with DISFA images are converted to 3D FLAME meshes using EMOCA (Li et al., 16 Jul 2025).

AUCodebook is implemented in PyTorch and trained on an RTX 6000 Ada GPU. Its Transformer has 8 layers, latent dimension 1024, learning rate $$\mathbf{V}(\mathbf{a}) = \mathbf{V}_0 + \sum_{k=1}^{N} a_k \mathbf{B}_k,$$8, batch size 1, and 200 training epochs, optimized with Adam. StyleBlendNet uses hidden and latent dimension 1024, 8 attention heads, 8 Transformer layers, learning rate $$\mathbf{V}(\mathbf{a}) = \mathbf{V}_0 + \sum_{k=1}^{N} a_k \mathbf{B}_k,$$9, batch size 1, and 400 epochs, also with Adam. No explicit data augmentation is described beyond the variety of identities and AU combinations.

At inference time, the pipeline is split into a one-time rig generation stage and a low-cost expression synthesis stage. For a new identity mesh $\mathbf{B}_k \in \mathbb{R}^{V \times 3}$0, StyleBlendNet and the AUCodebook decoder generate predicted AU-Blendshape bases $\mathbf{B}_k \in \mathbb{R}^{V \times 3}$1. Expression control is then obtained by linear blending:

$\mathbf{B}_k \in \mathbb{R}^{V \times 3}$2

The reported runtime is 0.3 seconds on RTX 6000 for auto-rigging an identity and approximately 0.002 seconds for generating many expressions through linear blending. By contrast, a CK+ or D3DFACS-style workflow without an AU-Blendshape rig requires separate neural inference for each AU configuration, yielding 2.7 seconds for seven emotional expressions.

The practical requirements are narrowly defined. The mesh topology must be consistent with FLAME, or a retargeting step must be introduced. AU annotations are not required for a new identity; only the neutral mesh is needed to auto-generate the AU-Blendshape rig. The source code is available at https://github.com/wslh852/AUBlendNet.git.

5. Empirical evaluation and demonstrated uses

The principal evaluation for stylized expression control uses mean squared error under single-AU and multi-AU settings. On AUBlendSet’s test set, training on AUBlendSet yields $\mathbf{B}_k \in \mathbb{R}^{V \times 3}$3 and $\mathbf{B}_k \in \mathbb{R}^{V \times 3}$4, while models trained on D3DFACS and CK+ show higher errors (Li et al., 16 Jul 2025). The qualitative interpretation in the paper is that models trained from direct AU labels struggle with multi-AU combinations not seen during training and can produce spurious motions or missing detail, whereas AUBlendSet-based rig prediction yields accurate, consistent AU-specific movements with smooth handling of arbitrary AU combinations.

The work also compares AUBlendNet with alternative model classes applied to the same rig prediction problem. Using the same $\mathbf{B}_k \in \mathbb{R}^{V \times 3}$5 and $\mathbf{B}_k \in \mathbb{R}^{V \times 3}$6 criteria, AUBlendNet achieves the best reported values: 4.55 and 7.62 (both scaled by $\mathbf{B}_k \in \mathbb{R}^{V \times 3}$7), compared with 5.66 and 8.71 for a diffusion model, 8.48 and 12.79 for a VQ-VAE model, and substantially worse results for a plain Transformer. In a user study involving 32 general users and 8 animation experts, AUBlendNet is preferred over an ARKit-based universal blendshape rig and a one-hot-label-based emotional generation workflow on manipulation freedom, interactiveness, generation satisfaction, style consistency, and response time.

A second application is speech-driven emotional facial animation. The paper decomposes facial motion into a neutral template, a speech-driven lip-and-jaw component, and an AU-driven emotion component:

$\mathbf{B}_k \in \mathbb{R}^{V \times 3}$8

In this arrangement, an existing speech model supplies $\mathbf{B}_k \in \mathbb{R}^{V \times 3}$9, while AUBlendNet supplies identity-specific AU-Blendshape offsets for emotion. The evaluation uses LVE, V-LVE, FDD, and Diversity on MEAD and VOCASET. The reported finding is that LVE increases slightly because some AUs affect mouth corners, but V-LVE and FDD remain stable, while Diversity increases for all three speech models on MEAD and becomes nonzero on VOCASET, which originally has no emotion labels.

A third application is data augmentation for AU recognition. AUBlendNet-generated rigs are used to synthesize additional AU combinations and style variations for training AU detectors. On D3DFACS and DISFA, the paper reports higher average F1 over eight AUs for all three evaluated detectors. The exact improvements are: HMP-PS from 55.04 to 57.96 on D3DFACS and from 61.96 to 64.57 on DISFA; LP-Net from 52.76 to 54.96 on D3DFACS and from 56.61 to 59.35 on DISFA; and SRERL from 52.48 to 55.96 on D3DFACS and from 55.75 to 56.32 on DISFA. This suggests that the learned AU-Blendshape prior is sufficiently structured to support recognition-oriented synthesis, not only animation.

6. Limitations, interpretive significance, and nomenclature

The method has several explicit constraints. It relies on FLAME topology with 5,023 vertices, so applying it to arbitrary meshes requires retargeting or fitting to FLAME (Li et al., 16 Jul 2025). Its style space is bounded by identities sampled from FLAME’s shape and expression space, with ages from 20 to 60; very extreme cartoon styles or out-of-distribution morphologies may therefore be only partially represented. AUBlendSet itself required extensive artist labor to create 32 AU-Blendshape control bases for each of 500 identities, which makes scaling to more identities or substantially different style regimes costly. The paper also notes that performance on noisy or unconstrained scans may degrade, despite partial mitigation through EMOCA-based fitting.

From a systems perspective, AUBlendNet’s significance lies in its separation of semantically interpretable control from identity-specific geometry adaptation. AUs serve as continuous scalar controls rather than discrete emotion labels, and the network’s role is to generate the per-identity basis that makes those controls meaningful for a particular face. This suggests a useful abstraction layer between high-level affective signals and low-level mesh deformation.

A terminological ambiguity should be noted. In another paper, "The Alpha Blending Hypothesis: Compositing Shortcut in Deepfake Detection" (Yermakov et al., 11 May 2026), the name “AUBlendNet” is mentioned only hypothetically to describe a detector centered on alpha-blending artifacts, while the actual method introduced there is BlenD. That usage is unrelated to the 3D facial animation framework defined in (Li et al., 16 Jul 2025). In established usage within the cited literature, AUBlendNet denotes the network for continuous 3D facial expression manipulation through facial AUs, not the compositing-focused deepfake detector.