AU Codes, Language, and Synthesis: Translating Anatomy to Text for Facial Behavior Synthesis

Abstract: Facial behavior synthesis remains a critical yet underexplored challenge. While text-to-face models have made progress, they often rely on coarse emotion categories, which lack the nuance needed to capture the full spectrum of human nonverbal communication. Action Units (AUs) provide a more precise and anatomically grounded alternative. However, current AU-based approaches typically encode AUs as one-hot vectors, modeling compound expressions as simple linear combinations of individual AUs. This linearity becomes problematic when handling conflicting AUs--defined as those which activate the same facial muscle with opposing actions. Such cases lead to anatomically implausible artifacts and unnatural motion superpositions. To address this, we propose a novel method that represents facial behavior through natural language descriptions of AUs. This approach preserves the expressiveness of the AU framework while enabling explicit modeling of complex and conflicting AUs. It also unlocks the potential of modern text-to-image models for high-fidelity facial synthesis. Supporting this direction, we introduce BP4D-AUText, the first large-scale text-image paired dataset for complex facial behavior. It is synthesized by applying a rule-based Dynamic AU Text Processor to the BP4D and BP4D+ datasets. We further propose VQ-AUFace, a generative model that leverages facial structural priors to synthesize realistic and diverse facial behaviors from text. Extensive quantitative experiments and user studies demonstrate that our approach significantly outperforms existing methods. It excels in generating facial expressions that are anatomically plausible, behaviorally rich, and perceptually convincing, particularly under challenging conditions involving conflicting AUs.

• The paper introduces AU descriptions, BP4D-AUText, AAAD, and VQ-AUFace to represent interacting facial muscle actions through natural language rather than one-hot AU labels.
• VQ-AUFace achieves the best LPIPS (0.390) and AAAD (0.606), while its FID (63.604) trails GANimation’s 29.388, indicating stronger behavioral alignment but weaker raw image quality.
• The text representation is central to the gains: replacing AU descriptions with one-hot labels reduces AAAD from 0.606 to 0.459, while VQ-AUFace earns the highest conflict-handling score (4.102) in a 15-person user study.

Motivation and problem formulation

This paper addresses a specific deficiency in controllable face synthesis: the inability of existing control signals to represent interactions among simultaneously activated facial muscles. Emotion-category-driven text-to-face models collapse behaviorally distinct states (e.g., melancholy, distress, and suppression under the umbrella of "sadness") into coarse stereotypes. AU-label-driven models preserve anatomical granularity but encode Action Units (AUs) as one-hot vectors and treat compound expressions as linear superpositions. The authors identify conflicting AUs—pairs such as AU15 and AU17 that act antagonistically on the same muscle group—as the failure case where this linearity produces anatomically implausible artifacts, since no mechanism exists to resolve biomechanical competition between opposing actions.

The proposed remedy is to replace discrete AU labels with natural language "AU Descriptions" as the control signal. Text can express joint physiological effects of interacting AUs (e.g., that AU15's downward pull is partially offset by AU6/AU12 when co-activated), something an orthogonal label vector cannot encode. The paper contributes four artifacts: the task formulation itself, the BP4D-AUText dataset, the AAAD evaluation metric, and VQ-AUFace as a baseline instantiation.

BP4D-AUText dataset

BP4D-AUText is constructed from 302,169 images drawn from BP4D and BP4D+, each containing at least one activated AU among 12 individual AUs. A rule-based Dynamic AU Text Processor converts AU annotations into fluent descriptions following three principles: canonical FACS definitions for single AUs, concatenation for non-conflicting combinations, and integrated phrases derived from Ekman's documented Appearance Changes for 26 interacting AU combinations—38 constructs total, each with five GLM-generated paraphrases for lexical diversity, with manual verification on a subset.

Two statistics are notable. First, conflicting AU combinations account for 81.4% of all entries (246,056 images) with 408,376 total occurrences, meaning images frequently contain multiple conflicting pairs. This supports the authors' claim that conflict resolution is not a corner case but a requirement for modeling spontaneous expressions, and it directly undermines the adequacy of linear AU superposition for most real-world data. Second, at 50–600 tokens per description, the text length exceeds what CLIP text encoders accommodate, motivating the substitution of T5-large in the synthesis pipeline. Reference faces are selected as neutral (all-AU-inactive) images per identity where available.

A limitation worth noting: the dataset inherits the demographic and recording conditions of BP4D/BP4D+ (posed/spontaneous lab sessions), and coverage is restricted to the 12 AUs annotated in those corpora; generalization beyond these AUs is untested within this work.

VQ-AUFace architecture

VQ-AUFace operates in two stages. Stage I trains a Residual VQGAN (inspired by VQGAN and RQ-VAE) for discrete facial image tokenization, combining reconstruction, commitment, hinge adversarial, VGG16 perceptual, and a novel facial anatomical loss computed on MEFARG features from ME-GraphAU. The anatomical loss is withheld early in training until reconstructions exhibit basic facial structure—a practical concession to instability. Stage II performs generation: a Face Encoder extracts appearance tokens ($T_A$, via face recognition and CLIP encoders), structure tokens ($T_S$), and VAE tokens ($T_V$); T5-large encodes AU Descriptions into $T_T$; cross-attention between $T_T$ and $T_S$ yields AU tokens ($T_{AU}$) encoding spatially coherent muscle activation patterns; a weighted fusion of $T_A$ and $T_{AU}$ concatenated with $T_V$ conditions a MaskGIT-style Transformer that autoregressively generates discrete image tokens decoded by the Residual VQGAN. Training used eight RTX 3090 GPUs for roughly 48 hours.

Evaluation methodology

The paper proposes AAAD (Alignment Accuracy of AU Probability Distributions), motivated by the absence of any semantic-consistency metric designed for facial data. AAAD computes cosine similarity between AU activation probability vectors predicted from the generated image (via ME-GraphAU, the top performer on BP4D) and from the input text (via a T5-large + MLP classifier trained on synthetic description–label pairs, F1 = 0.731), then normalizes against maximum similarity over ground-truth pairs and minimum over shuffled pairs. Two caveats apply: the metric covers only 12 AUs, and its reliability is bounded by the 0.731 F1 of the text-side classifier—an error source the paper does not propagate into its reported scores.

Quantitative results

Against AnyFace, GANimation (retrained on BP4D-AUText), UniPortrait (inference-only), and LoRA-fine-tuned Stable Diffusion 1.5, VQ-AUFace achieves the best LPIPS (0.390) and best AAAD (0.606), with Stable Diffusion second on AAAD (0.602). GANimation, despite receiving ground-truth AU labels, attains lower AAAD than both text-conditioned methods—an empirical point in favor of the central hypothesis that textual AU descriptions carry more behavioral semantics than labels. The comparison is not uniformly favorable: GANimation achieves by far the best FID (29.388 vs. 63.604 for VQ-AUFace), so the claimed advantage is specifically in behavioral/semantic fidelity rather than raw distributional image quality. Also note the asymmetric comparison protocol: UniPortrait and AnyFace were never fine-tuned on BP4D-AUText, which disadvantages them on domain-specific metrics.

Ablations

Token ablations across ten trained variants show that removing $T_S$0 degrades both quality and consistency most severely, removing $T_S$1 mainly hurts AAAD (0.574 vs. 0.606), and removing $T_S$2 has minimal effect—consistent with $T_S$3 carrying only high-level identity information. Models trained with the anatomical loss achieve better AAAD overall, supporting its role in enforcing biomechanical plausibility, though the w/o-anatomical-loss variants without $T_S$4/$T_S$5 actually attain better FID/KID, indicating the loss trades some distributional quality for semantic alignment.

The most decisive ablation replaces AU text with one-hot AU labels projected through an MLP under otherwise identical architecture and training: AAAD collapses from 0.606 to 0.459, with all image-quality metrics deteriorating substantially. Because the architecture is held fixed, this isolates the input representation as the causal factor, directly substantiating the paper's core claim rather than leaving it confounded with model design.

Generalization is evaluated on 2,000 unseen AU combinations, where performance matches or slightly exceeds the in-distribution test set (AAAD 0.607, FID 62.180). The authors attribute this to the model learning per-AU anatomical constraints from text rather than memorizing combination-specific mappings—a plausible reading, though the unseen combinations are still recompositions of the same 12 AUs, so this tests combinatorial rather than cross-AU generalization.

Conflict handling user study

Fifteen participants ranked outputs from five systems on six conflicting AU combinations using FACS-derived visual criteria, scored as the AU Conflict Handling Score (AUCHS). VQ-AUFace achieved the highest mean score (4.102), ahead of GANimation (3.258), UniPortrait (3.174), Stable Diffusion (2.860), and AnyFace (1.575). The margin is largest on multi-way conflicts (e.g., AU6+AU12+AU15+AU17: 4.384 vs. 2.294 for SD). One exception is acknowledged: GANimation outperforms VQ-AUFace on AU1+AU4 (3.814 vs. 3.506), showing the advantage is consistent but not universal. Qualitative comparisons further show baselines producing truncated faces, magenta color casts, and unresolved antagonisms (e.g., GANimation rendering downward mouth corners despite raised cheeks under AU6+AU12+AU15).

Limitations and open questions

Several constraints bound the results. The paradigm is validated only through one baseline architecture; the authors themselves state VQ-AUFace is "just one possible instantiation" and that integration with modern diffusion-based text-to-image backbones remains unexplored here. AAAD depends on learned AU recognizers whose own errors are unquantified in the final metric, and it spans only 12 AUs of the FACS taxonomy. The dataset derives from two related corpora with overlapping collection protocols, and the 26 interacting-combination templates were authored manually from the FACS manual, so coverage of interaction types outside those 26 defaults to concatenation. Finally, the generalization experiment is confined to recombinations of seen AUs; whether the approach transfers to AUs absent from training or to cross-dataset demographics is left open.

Conclusion

This paper reframes facial behavior synthesis around linguistically expressed AU semantics, supplying the task definition, a large paired dataset dominated by conflicting AU cases, a controlled ablation demonstrating that the textual representation itself—not architecture—drives the gains, and a purpose-built consistency metric. The evidence supports the claim that AU Descriptions resolve muscular antagonisms that defeat linear label encodings, though the advantage is specific to behavioral fidelity rather than unconditional image quality, and validation rests on a single baseline within a bounded AU vocabulary.