FACS-based Blendshape Representation

• The paper introduces a semantically factorized model where each blendshape channel aligns with specific FACS action units, enabling intuitive control over localized facial movements.
• It employs a linear blendshape formulation that combines a neutral mesh with additive deformation vectors to capture precise actions like jaw drop, brow raise, and eye blinks.
• Evaluations demonstrate significant improvements in rig automation, cross-identity retargeting, and animation fidelity, highlighting practical benefits for real-time applications.

Searching arXiv for the cited papers and closely related work on FACS-based blendshape representations. FACS-based blendshape representation is a facial parameterization in which a neutral mesh is combined with a set of semantically meaningful deformation channels aligned with Facial Action Coding System (FACS) action units or closely related animation controls. In its classical form, the face is written as a neutral shape plus weighted blendshape deltas, so that individual channels correspond to localized actions such as jaw open, lip corner pull, brow raise, blink, or cheek motion. Across recent systems, this representation appears in several concrete instantiations rather than as a single universal basis: 52 ARKit-compatible channels for monocular mobile tracking, 53 ICT-FaceKit expression blendshapes for mesh-agnostic cloning and inverse rendering, 32 AU-specific basis vectors in AUBlendSet, 55 FACS-based blendshapes in ICT-FaceKit-based talking-head synthesis, and up to 155 shapes in fully automatic rigging pipelines (Grishchenko et al., 2023, Cha et al., 28 May 2025, Li et al., 16 Jul 2025, Park et al., 28 Jul 2025, Wang et al., 6 Jun 2026).

1. Core formulation and control semantics

The canonical mathematical form is the linear blendshape model. One explicit formulation uses a neutral mesh $\mathbf{b_0}$, fully activated blendshape meshes $\mathbf{b_i}$, and scalar activations $w_i\in[0,1]$:

$$\mathbf{b} = \mathbf{b_0} + \sum_{i=1}^{52} w_i (\mathbf{b_i} - \mathbf{b_0}).$$

A closely related form appears in automatic rigging systems as

$$\mathbf{v}(\mathbf{w}) = \mathbf{v}_0 + \sum_{i=1}^N w_i \,\Delta \mathbf{v}_i,$$

with $\mathbf{v}_0$ the neutral mesh, $\Delta \mathbf{v}_i$ the per-shape displacement field, and $N$ the number of channels (Grishchenko et al., 2023, Wang et al., 6 Jun 2026).

The defining property of the FACS-based variant is not merely linearity but semantic factorization. Each channel is intended to encode an interpretable local movement rather than an arbitrary PCA mode. In the ARKit-compatible 52-channel setting, this includes jaw and mouth movements, cheek and nose actions, eyelid and eye squint or widen, brow raise and lower, and lip roll, press, and stretch; the representation is explicitly described as strongly inspired by FACS-style action units (Grishchenko et al., 2023). In AUBlendSet, the formulation becomes even more direct: one AU corresponds to one blendshape basis vector, and a continuous AU coefficient vector $\mathbf{a}$ synthesizes an expression as

$$\mathbf{M}_{\text{expr}} = \mathbf{M}_0 + \sum_{i=1}^{N} a_i \mathbf{B}_i,$$

with $\mathbf{b_i}$0 retained AUs on FLAME topology (Li et al., 16 Jul 2025).

A recurrent distinction in the literature is between semantically authored FACS-style controls and data-driven expression bases. Several works explicitly retain artist-designed or AU-indexed channels because they are easier to constrain, easier to retarget, and more interpretable for animation than latent coordinates. One formulation states this directly: animators want to adjust “smile” rather than “mode 3” (Grishchenko et al., 2023). By contrast, statistical slider systems such as SliderGAN use sparse-PCA expression components and continuous parameters in $\mathbf{b_i}$1; these are blendshape sliders, but not explicit FACS channels (Ververas et al., 2019).

2. Relation to FACS, ARKit, and AU taxonomies

A frequent misconception is that “FACS-based” denotes a single fixed basis. The cited systems instead define several semantically related control spaces, with different cardinalities and different degrees of explicit AU correspondence.

System	Control space	Relation to FACS
Blendshapes GHUM	52 blendshapes	ARKit-compatible, strongly FACS-inspired
ICT-FaceKit-based systems	53 or 55 blendshapes	FACS-style expression controls
AUBlendSet / AUBlendNet	32 AUs	One AU = one blendshape basis vector
OmniFaceRig	up to 155 shapes	Canonical FACS set with side-specific variants and correctives
RigAnyFace	48 primary FACS poses + 48 correctives	Industry-standard FACS pose library

The 52-channel ARKit-compatible design in Blendshapes GHUM is explicitly chosen because it is familiar to 3D studios and animators, while remaining close to FACS semantics such as AU12-like smiling, AU1-like inner brow raise, AU26-like jaw drop, and AU5-like upper lid raise (Grishchenko et al., 2023). AUBlendSet makes the AU correspondence explicit by starting from the standard FACS AU set, removing AUs related to head pose and tongue, and merging some highly similar eye AUs, yielding 32 retained units; each character then receives 32 AU-Blendshape basis vectors on a 5,023-vertex FLAME mesh (Li et al., 16 Jul 2025).

ICT-FaceKit occupies an intermediate position. In mesh-agnostic cloning and talking-head synthesis, it supplies 53 or 55 expression blendshapes whose labels are FACS-like and semantically localized, including eye, brow, cheek, jaw, and mouth controls (Cha et al., 28 May 2025, Park et al., 28 Jul 2025). In JOLT3D, these 55 blendshapes are the main expression parameterization for both reconstruction and lip-sync; 35 of them are designated as mouth-related and can be replaced independently for audio-driven modification (Park et al., 28 Jul 2025).

At the large-rig end, OmniFaceRig organizes its output into Core, Additional, and Full configurations: approximately 13 basic controls, approximately 46 higher-resolution controls, and a Full set of 155 shapes with side-specific variants and correctives. The paper does not publish a full AU-to-shape table, but it explicitly describes the result as a FACS-based rig and situates it in the space of production-style brow, eyelid, nose, mouth, jaw, viseme-like, and corrective controls (Wang et al., 6 Jun 2026). RigAnyFace similarly targets 48 primary FACS poses and 48 corrective poses as artist-authored blendshapes (Ma et al., 23 Nov 2025).

3. Acquisition, fitting, and personalization of FACS-style blendshapes

One major line of work obtains FACS-style coefficients by fitting a semantic rig to registered 3D data. Blendshapes GHUM uses a lab setup with calibrated multi-camera capture at 60 Hz, scanning 6,000 identities, each performing 40 predefined expressions chosen so that every blendshape is activated in at least one clip. Scans are registered to a canonical 12,201-vertex template using 478 facial landmarks and an as-conformal-as-possible procedure; a technical artist’s 52 canonical blendshapes are then transferred to each subject’s neutral mesh by affine deformation transfer, and per-frame coefficients are optimized with L-BFGS together with a global rigid transform. The result is a 52-dimensional coefficient vector with ARKit/FACS semantics for every frame, obtained without manual coefficient annotation (Grishchenko et al., 2023).

A second line of work personalizes a fixed FACS-style library from minimal input. “Dynamic Facial Asset and Rig Generation from a Single Scan” assumes a generic template rig of 55 additive blendshape displacement fields whose naming convention follows Apple ARKit and includes additional asymmetric eyebrow shapes. It constructs 26 predefined FACS expressions as binary combinations of these 55 units, then learns subject-specific offsets $\mathbf{b_i}$2 from a single neutral scan so that personalized blendshapes satisfy

$\mathbf{b_i}$3

and subject expressions are reconstructed by

$\mathbf{b_i}$4

A two-stage self-supervised scheme first estimates personalized offsets under locality-aware regularization and then refines both offsets and blending weights to account for unintended motions in real FACS performances (Li et al., 2020).

AUBlendSet provides a more explicit AU library. It contains 500 identities, each with a neutral template mesh and 32 AU-Blendshape basis vectors authored by professional artists using FACS exemplar images and descriptions. The dataset also includes AU-annotated facial postures with continuous intensities rather than only binary or discrete labels. Because all meshes share FLAME topology, the learned basis vectors are directly comparable across characters and can be used to train AU-conditioned generators such as AUBlendNet (Li et al., 16 Jul 2025).

These pipelines share a common structural assumption: identity is stored in the neutral mesh and expression is stored in semantically indexed deformation fields. The personalization mechanisms differ—offline optimization from scans, self-supervised prediction from neutral geometry, or direct artist authoring—but all preserve the channel identity of the original FACS-style template (Grishchenko et al., 2023, Li et al., 2020, Li et al., 16 Jul 2025).

4. Neural, anatomical, and implicit reformulations

Recent work increasingly preserves the FACS-style control interface while changing the internal deformation model. In “Anatomically Constrained Implicit Face Models,” the public interface remains a jaw transform, expression weights $\mathbf{b_i}$5, and optional head transform, but the neutral surface and per-expression correctives are represented by SIREN-based implicit fields rather than stored vertex deltas. The deformed skin is written as

$\mathbf{b_i}$6

so the model remains formally analogous to a jaw-rigged blendshape rig while adding dense anatomical constraints derived from a learned internal anatomy surface (Chandran et al., 2023).

“High-Quality Mesh Blendshape Generation from Face Videos via Neural Inverse Rendering” retains a standard linear expression model,

$\mathbf{b_i}$7

with $\mathbf{b_i}$8 semantically labeled ICT expression blendshapes, but reparameterizes per-vertex deformation through differential coordinates over a tetrahedral mesh. Locality, sparsity, and symmetry regularizers are used specifically to stop optimized personalized bases from losing AU-like semantics during joint inverse rendering of geometry, appearance, and motion (Ming et al., 2024).

“Neural Face Skinning for Mesh-agnostic Facial Expression Cloning” makes the FACS supervision explicit at the latent-code level. It uses ICT-FaceKit with 53 expression blendshapes and supervises the first 53 dimensions of a 128-dimensional global expression code $\mathbf{b_i}$9 to match the ICT expression weights:

$w_i\in[0,1]$0

A per-vertex skinning encoder then localizes this global FACS-interpretable code by predicting vertex-wise weights $w_i\in[0,1]$1 and forming localized codes

$w_i\in[0,1]$2

which drive a decoder on arbitrary target meshes, including stylized ones (Cha et al., 28 May 2025).

A different reformulation is compression rather than replacement. “Compressed Skinning for Facial Blendshapes” treats the input rig—often a large FACS-derived blendshape system—as a black box and bakes it into proxy-bone linear blend skinning. Starting from the classical delta form

$w_i\in[0,1]$3

it learns sparse skinning weights and sparse per-blendshape transform coefficients so that runtime evaluation can use a small number of affine bone transforms instead of hundreds of dense shapes. The FACS control semantics remain unchanged; only the geometric realization is compressed (Kavan et al., 2024).

5. Retargeting, stylization, talking heads, and speech-driven control

Because the channels are semantically labeled, FACS-based blendshape representations are especially suited to cross-identity control and modality transfer. AUBlendNet predicts, from a neutral identity mesh alone, the full set of 32 AU-Blendshape basis vectors for that character. It does this by decoding a style-aware latent code through a learned AUCodebook and then applying the standard linear AU synthesis rule. On AUBlendSet, the reported errors for single-AU and multi-AU control are $w_i\in[0,1]$4 and $w_i\in[0,1]$5, respectively, and the resulting rigs are used for stylized expression manipulation, speech-driven emotional animation, and AU-recognition data augmentation (Li et al., 16 Jul 2025).

JOLT3D uses the 55 ICT-FaceKit FACS-based blendshapes as the expression subspace of a jointly trained reconstruction-and-generation system for talking heads. ReconNet predicts identity $w_i\in[0,1]$6, blendshape coefficients $w_i\in[0,1]$7, head pose, and eye pose from RGB frames; a feature-warping GAN then conditions on rendered sketches and 3DMM-induced flow fields. The same FACS-based $w_i\in[0,1]$8 is also the target space for an audio-to-expression diffusion model that outputs 35 mouth-related blendshape coefficients. Because the blendshape representation is semantically localized, JOLT3D can replace only the mouth subset of $w_i\in[0,1]$9 during lip-sync and preserve the non-mouth channels, which is used to decouple the original chin contour from the lip-synced chin contour and reduce flickering near the mouth (Park et al., 28 Jul 2025).

Work on speech-expression disentanglement reaches a similar structural decomposition even when it does not explicitly use FACS labels. “Learning Disentangled Speech- and Expression-Driven Blendshapes for 3D Talking Face Animation” models deformation as

$$\mathbf{b} = \mathbf{b_0} + \sum_{i=1}^{52} w_i (\mathbf{b_i} - \mathbf{b_0}).$$0

with a sparsity loss on the cross-domain coefficients $$\mathbf{b} = \mathbf{b_0} + \sum_{i=1}^{52} w_i (\mathbf{b_i} - \mathbf{b_0}).$$1 and $$\mathbf{b} = \mathbf{b_0} + \sum_{i=1}^{52} w_i (\mathbf{b_i} - \mathbf{b_0}).$$2. The paper does not define AUs, but it explicitly notes that the learned expression blendshapes are conceptually close to a FACS-style rig, particularly as a compositional upper-face and affective expression layer that can be added to a speech-driven mouth layer. This suggests a direct route to AU-conditioned talking-face systems by replacing or supervising $$\mathbf{b} = \mathbf{b_0} + \sum_{i=1}^{52} w_i (\mathbf{b_i} - \mathbf{b_0}).$$3 with a FACS-aligned basis (Mao et al., 29 Oct 2025).

Retargeting across topology is also an important theme. Mesh-agnostic neural face skinning localizes a shared FACS-based global code on arbitrary target meshes by indirect supervision from anatomical segmentation labels, while RigAnyFace and OmniFaceRig generate full production-style FACS rigs automatically for previously unrigged assets (Cha et al., 28 May 2025, Ma et al., 23 Nov 2025, Wang et al., 6 Jun 2026).

6. Automation, evaluation, and open limitations

Fully automatic rig generation extends FACS-based representation beyond captured human heads. OmniFaceRig converts a static surface-only 3D character mesh into an inner-mouth-aware FACS rig with up to 155 blendshapes, procedurally fitted teeth, gums, and tongue, repacked UVs, and collision-aware transfer across humans, humanoids, long-muzzled animals, and short-muzzled animals. Its Full set reaches approximately 155 shapes, and Omni-Bench provides 1,000 generated characters with complete FACS rigs and inner-mouth geometry. On screened Omni-Bench inputs, the reported metrics include MAE around $$\mathbf{b} = \mathbf{b_0} + \sum_{i=1}^{52} w_i (\mathbf{b_i} - \mathbf{b_0}).$$4–$$\mathbf{b} = \mathbf{b_0} + \sum_{i=1}^{52} w_i (\mathbf{b_i} - \mathbf{b_0}).$$5 mm, Q95 around $$\mathbf{b} = \mathbf{b_0} + \sum_{i=1}^{52} w_i (\mathbf{b_i} - \mathbf{b_0}).$$6–$$\mathbf{b} = \mathbf{b_0} + \sum_{i=1}^{52} w_i (\mathbf{b_i} - \mathbf{b_0}).$$7 mm, penetration rate around $$\mathbf{b} = \mathbf{b_0} + \sum_{i=1}^{52} w_i (\mathbf{b_i} - \mathbf{b_0}).$$8–$$\mathbf{b} = \mathbf{b_0} + \sum_{i=1}^{52} w_i (\mathbf{b_i} - \mathbf{b_0}).$$9, and success rate up to 99% on screened human and humanoid assets (Wang et al., 6 Jun 2026).

RigAnyFace similarly targets scalable neural auto-rigging to industry-standard FACS poses. It learns to deform arbitrary neutral meshes into 48 primary FACS poses plus 48 corrective poses, including assets with multiple disconnected components such as eyeballs. A key contribution is 2D supervision on unrigged neutral meshes, which augments a much smaller set of artist-rigged 3D heads and improves generalization to varied topologies while preserving a classical linear blendshape output compatible with existing DCC tools (Ma et al., 23 Nov 2025).

Evaluation criteria in this literature reflect both semantics and geometry. Blendshapes GHUM reports Mean Normalized Error for landmarks, with the real-time blendshape model’s reconstructed landmarks at 3.88% and fully activated canonical blendshape differences around 8.45% MNE, emphasizing projected landmark similarity rather than coefficient accuracy alone (Grishchenko et al., 2023). AIM reports average 3D fitting error over 819 frames and 5 actors, with 0.312 mm for AIM versus 0.095 mm for ALM and much larger errors for global and patch blendshape baselines, framing the trade-off between anatomical realism, speed, and strict fidelity (Chandran et al., 2023). AUBlendNet, JOLT3D, and the mesh-agnostic cloning literature supplement geometric errors with user studies, inverse-rigging metrics, lip-sync metrics, and per-segment evaluations (Li et al., 16 Jul 2025, Park et al., 28 Jul 2025, Cha et al., 28 May 2025).

Several limitations recur. One is that semantic alignment does not guarantee a universal basis: some systems are actor-specific, as in AIM, while others are tied to a particular template such as ARKit, ICT-FaceKit, or FLAME (Chandran et al., 2023, Grishchenko et al., 2023, Cha et al., 28 May 2025). A second is that many pipelines are only FACS-inspired rather than explicitly AU-supervised; SliderGAN, Blendshapes GHUM, and some inverse-rendering systems preserve semantic locality without enforcing a one-to-one AU ontology (Ververas et al., 2019, Grishchenko et al., 2023, Ming et al., 2024). A third is that the linear blendshape model, even when enhanced with correctives, dynamic textures, or compression, remains an approximation of nonlinear muscle-skin interactions. This is why anatomical simulation, implicit anatomy fields, and collision-aware inner-mouth modeling continue to appear as complementary directions rather than replacements for FACS semantics (Bao et al., 2018, Chandran et al., 2023, Wang et al., 6 Jun 2026).

Taken together, the recent literature treats FACS-based blendshape representation less as a single file format than as a stable semantic interface. The neutral-plus-deltas formulation remains central, but it now serves as the control layer for annotation-free monocular tracking, self-supervised personalization from a single scan, inverse-rendered rig reconstruction from video, anatomically constrained implicit models, mesh-agnostic neural retargeting, talking-head synthesis, automatic multi-species rigging, and compressed runtime deployment (Grishchenko et al., 2023, Li et al., 2020, Ming et al., 2024, Cha et al., 28 May 2025, Park et al., 28 Jul 2025, Wang et al., 6 Jun 2026).