AUBlendSet: 3D Facial Expression Dataset

• AUBlendSet is a 3D facial expression dataset featuring per-identity, artist-defined AU blendshape bases and annotated key poses for continuous, real-time editing.
• The dataset comprises 500 identities using a consistent FLAME topology with 5,023 vertices and 32 semantically meaningful action units to ensure precise control.
• Its companion network, AUBlendNet, delivers remarkable accuracy improvements (e.g., 4.55e-9 MSE) and faster inference (0.3 s) over traditional methods.

Searching arXiv for the specified paper to ground the article and confirm citation details. AUBlendSet is a 3D facial dataset based on AU-Blendshape representation for fine-grained facial expression manipulation across identities. It was introduced with AUBlendNet in “AU-Blendshape for Fine-grained Stylized 3D Facial Expression Manipulation” as, to the authors’ knowledge, the first publicly available 3D facial-expression dataset whose per-identity control space is not “universal” but sculpted in the artists’ workshop to exactly match each character’s style (Li et al., 16 Jul 2025). The dataset couples artist-defined action-unit blendshapes with annotated facial postures, and the companion network predicts AU-Blendshape basis vectors for a given identity mesh, enabling continuous, per-identity, action-unit driven editing in real time.

1. Dataset scope and composition

AUBlendSet is organized around 500 distinct character templates (“themes”) generated by sampling gender, age and facial-shape parameters in the FLAME model (Li et al., 16 Jul 2025). Of these 500 identities, 206 are female and 294 are male, and ages span roughly 20–60 years. All characters share the same expressive FLAME topology with 5,023 vertices, fixed connectivity as a triangular mesh, and per-vertex correspondence across identities.

The dataset is built from 32 standard facial action units. Tongue and head-pose AUs are dropped, and redundant eye-AUs are merged, leaving 32 independent, semantically meaningful basis units, including AU 1 Inner Brow Raiser, AU 4 Brow Lowerer, AU 6 Cheek Raiser, AU 9 Nose Wrinkler, and AU 12 Lip Corner Puller (Li et al., 16 Jul 2025). Each AU controls the mesh offset produced by the corresponding facial muscle group.

Beyond the 32 artist-sculpted AU-blendshape bases, each theme includes a set of “key poses” in which combinations of AUs are activated at varying intensities. These poses are manually annotated with their exact AU-weights and are used both for quantitative evaluation and as reference exemplars (Li et al., 16 Jul 2025).

Component	Value	Role
Character templates	500	Identity coverage
Female / male	206 / 294	Demographic composition
Mesh vertices	5,023	Shared expressive FLAME topology
AU bases per identity	32	Per-identity control space
Additional postures	Key poses with exact AU-weights	Evaluation and reference

This composition situates AUBlendSet between categorical or 2D AU-label collections and universal 3D blendshape packs. The paper explicitly places it in relation to AffectNet, CK+, DISFA, ARKit, and FaceWarehouse, arguing that its distinguishing property is the availability of per-identity, artist-defined, continuous AU bases plus exemplar poses (Li et al., 16 Jul 2025).

2. Annotation protocol and per-identity control space

Every one of the 500 FLAME template heads was assigned 32 new blendshape targets, one for each AU (Li et al., 16 Jul 2025). Professional animators sculpted these blendshapes from canonical FACS exemplars until three criteria were satisfied: symmetry, fidelity to the semantic definition of the AU, and stylistic consistency with the character’s neutral template. The results were then peer-reviewed to ensure there were no stray artifacts or topology violations.

The annotation pipeline is therefore explicitly “manual plus review,” while retaining a common underlying expressive FLAME mesh so that all per-identity blendshapes remain strictly isomorphic (Li et al., 16 Jul 2025). This isomorphic structure is central: it preserves correspondence across identities while allowing the local deformation basis itself to be character-specific rather than universal.

A common misconception in 3D facial control is that a single universal blendshape basis is sufficient for all identities. AUBlendSet is constructed around the opposite premise: the per-identity control space is sculpted to match each character’s style. This means that semantic consistency is enforced at the AU level, but the geometric realization of a smile, brow raise, or nose wrinkle can vary with identity-specific facial morphology and artistic style. The dataset therefore treats stylization not as post hoc retargeting noise, but as part of the control representation itself (Li et al., 16 Jul 2025).

3. AU-Blendshape representation and mathematical formulation

In AUBlendSet, every expression is represented as a linear blend of the neutral mesh and 32 AU-basis offsets. For a given identity, the final vertex-coordinate vector is

$X(\mathbf{w}) = X_0 + \sum_{i=1}^{32} w_i B_i,$

where $X_0 \in \mathbb{R}^{3V}$ is the neutral-pose vertex array with $V=5023$ vertices, $B_i \in \mathbb{R}^{3V}$ is the artist-sculpted displacement vector of AU $i$, and $w_i \in [0,1]$ is its activation weight (Li et al., 16 Jul 2025).

Because the basis vectors $B_i$ are handcrafted, no further PCA or global subspace fitting is needed at data-collection time; each basis is already a “pure AU” (Li et al., 16 Jul 2025). This distinguishes the representation from pipelines that derive latent expression directions by factorization or manifold fitting after capture. Here, semantic disentanglement is imposed during authoring rather than estimated statistically.

In AUBlendNet’s codebook stage, the set $\{B_i\}_{i=1}^{32}$ for each character is passed through a VQ-VAE inspired by CodeTalker to learn a discrete latent prior, but this serves compression and style-library construction rather than re-derivation of the AU bases themselves (Li et al., 16 Jul 2025). No additional sparsity or nonnegativity constraints are applied beyond clamping $w_i \ge 0$.

This representation has two immediate implications. First, continuous control is native: the AU weights directly parameterize expression intensity. Second, semantically meaningful editing and stylized geometry are not in tension, because the semantic basis is fixed at the level of AU definition while the geometric offsets are per-identity. A plausible implication is that this structure is particularly suitable for pipelines that need both interpretability and identity-specific stylization.

4. AUBlendNet architecture and training regime

AUBlendNet consists of two modules: AUCodebook and StyleBlendNet (Li et al., 16 Jul 2025). AUCodebook performs offline, per-identity codebook learning. Its architecture is a VAE-style transformer with 8 layers, 8 attention heads, and latent dimension $D=1024$. The input is the $X_0 \in \mathbb{R}^{3V}$0 blendshape matrix $X_0 \in \mathbb{R}^{3V}$1. The encoder $X_0 \in \mathbb{R}^{3V}$2 produces latent codes $X_0 \in \mathbb{R}^{3V}$3, which are quantized to $X_0 \in \mathbb{R}^{3V}$4, and the decoder $X_0 \in \mathbb{R}^{3V}$5 reconstructs $X_0 \in \mathbb{R}^{3V}$6.

The AUCodebook loss is given per theme as

$X_0 \in \mathbb{R}^{3V}$7

with $X_0 \in \mathbb{R}^{3V}$8, trained using Adam with $X_0 \in \mathbb{R}^{3V}$9, batch size 1, for 200 epochs (Li et al., 16 Jul 2025).

StyleBlendNet performs fast per-mesh prediction. Its input is the neutral-template mesh $V=5023$0. A small TCN performs temporal fill-in, then $V=5023$1 projects to $V=5023$2. Style embedding is injected through $V=5023$3 and an AdaLN-Zero block, $V=5023$4, which provides per-identity style codes $V=5023$5 (Li et al., 16 Jul 2025). The backbone comprises 8 transformer layers with AdaLN-Zero conditioning, 8 heads, and hidden dimension 1024. The predicted latent codes $V=5023$6 are passed to the AUCodebook decoder to obtain $V=5023$7.

The synthesis loss is

$V=5023$8

and training uses Adam with $V=5023$9, batch size 1, for 400 epochs (Li et al., 16 Jul 2025).

The architectural decomposition separates style-library construction from inference-time prediction. AUCodebook learns a discrete latent prior over per-identity AU bases, while StyleBlendNet maps a neutral mesh to style-conditioned latent codes and decodes them to blendshape bases. This suggests that the network is designed not merely for expression transfer, but for direct prediction of the control basis itself.

5. Empirical validation across manipulation, animation, and recognition

For stylized 3D facial expression manipulation, the paper compares AUBlendNet trained on D3DFACS, CK+ converted to FLAME via EMOCA, and AUBlendSet, using the AUBlendSet test split with an 8:1:1 identity split (Li et al., 16 Jul 2025). Evaluation uses mean-squared vertex error under single-AU control, $B_i \in \mathbb{R}^{3V}$0, and multi-AU control, $B_i \in \mathbb{R}^{3V}$1.

Training data	$B_i \in \mathbb{R}^{3V}$2	$B_i \in \mathbb{R}^{3V}$3
D3DFACS	226.41	68.21
CK+	8490.27	2028.14
AUBlendSet	4.55	7.62

The same comparison reports inference times of 2.7 s for D3DFACS with 7 poses, 2.7 s for CK+, and 0.3 s for AUBlendSet (Li et al., 16 Jul 2025). The paper states that AUBlendNet on AUBlendSet outperforms all others by two orders of magnitude in accuracy and is over 8× faster because it predicts all 32 blendshapes in one pass, then blends in approximately 2 ms.

Against three other generative backbones trained on AUBlendSet—Latent Diffusion (FaceDiffuser), VQ-VAE, and a standard Transformer—AUBlendNet also yields the lowest reported errors: $B_i \in \mathbb{R}^{3V}$4 for single- and multi-AU control, compared with $B_i \in \mathbb{R}^{3V}$5 for diffusion, $B_i \in \mathbb{R}^{3V}$6 for VQ-VAE, and $B_i \in \mathbb{R}^{3V}$7 for the Transformer (Li et al., 16 Jul 2025).

A user study involving 32 lay users and 8 professional animators compared three editing paradigms on freedom of editing, interactivity, satisfaction, style consistency, and latency using a 5-point Likert scale. AUBlendNet scored highest on every measure, including editing freedom at 4.7/5 versus 3.9 for ARKit/blendshapes and 3.2 for one-hot emotion, and satisfaction at 4.8 versus 4.1 and 3.5 (Li et al., 16 Jul 2025).

The system is also evaluated in speech-driven emotional facial animation. Given speech-driven mouth offset $B_i \in \mathbb{R}^{3V}$8, the final mesh is formed as

$B_i \in \mathbb{R}^{3V}$9

On MEAD and VOCASET, AUBlendNet is injected into FaceFormer, FaceDiffuser, and LGLDM, and evaluated with Lip Vertex Error, Upper-face Dynamics Deviation, Velocity LVE, and expression Diversity (Li et al., 16 Jul 2025). On MEAD, LVE degrades slightly, exemplified by FaceFormer moving from $i$0 to $i$1, while FDD remains tight, exemplified by $i$2 to $i$3, V-LVE is unchanged, and Diversity increases by 25–40% across all three methods. On VOCASET, where no emotion labels are available, Diversity becomes non-zero at approximately $i$4, whereas the baselines had none (Li et al., 16 Jul 2025). The reported interpretation is that AUBlendNet adds controllable, personalized emotional nuance without desynchronizing lip sync.

A third application is emotion-recognition data augmentation. The method synthesizes diversified AU activations on 3D faces via AUBlendNet and retrains HMP-PS, LP-Net, and SRERL on D3DFACS and DISFA (Li et al., 16 Jul 2025). Average F1 improvements are reported as follows: HMP-PS from 55.04 to 57.96 on D3DFACS and 61.96 to 64.57 on DISFA; LP-Net from 52.76 to 54.96 and 56.61 to 59.35; SRERL from 52.48 to 55.96 and 55.75 to 56.32. The paper summarizes this as a 2–3 percentage-point absolute boost in F1 even on very small AU sets.

6. Significance, positioning, and prospective extensions

The paper presents AUBlendSet as filling a gap between purely categorical or 2D AU-label collections such as AffectNet, CK+, and DISFA, and universal 3D blendshape packs such as ARKit and FaceWarehouse (Li et al., 16 Jul 2025). Its central contribution is the combination of per-identity, artist-defined, continuous AU bases with exemplar poses. In that framing, the dataset provides an interactive, fine-grained, stylized 3D control interface for facial expression.

AUBlendNet extends this by making it practical to generate the blendshapes on the fly rather than hand-sculpting them for every new identity (Li et al., 16 Jul 2025). The paper states that new identities, including ones not in the original 500, can be incorporated with a few seconds of network inference. This suggests a shift from manually authored control rigs toward learned prediction of style-consistent AU bases under fixed topology constraints.

The downstream applications already demonstrated are stylized retargeting, speech-driven animation, and robust AU detection (Li et al., 16 Jul 2025). The paper identifies real-time avatar systems for VR/AR, personalized digital humans in games and film, and integration with full-body motion retargeters as natural next steps. It also proposes future extensions to dynamic 4D scans, implicit or neural-SDF based topologies, and semi-supervised or few-shot fine-tuning to automate the initial artist “sketch.”

The main conceptual significance of AUBlendSet is therefore not only the provision of another 3D face corpus, but the explicit redefinition of the facial control basis as identity-specific while remaining semantically aligned at the AU level. A plausible implication is that this formulation may be especially useful in settings where retargetability, stylization, and interpretability must coexist without collapsing into a single universal deformation basis.