- The paper introduces a fully automatic pipeline for converting static 3D meshes into comprehensive facial rigs with inner-mouth synthesis, achieving near-perfect rigging success (99%).
- It employs a hybrid VLM and a four-model segmentation ensemble to robustly detect facial features across human and animal topologies.
- The pipeline integrates topology-specific template registration and procedural inner-mouth geometry, reducing rigging time from manual hours to seconds.
OmniFaceRig: Fully Automatic Inner-Mouth-Aware Face Rigging Across Diverse 3D Character Topologies
Overview and Motivation
OmniFaceRig introduces a fully automatic pipeline for converting static surface-only 3D character meshes into comprehensive, inner-mouth-aware FACS facial rigs. The system targets the longstanding bottleneck in character animation: the manual authoring of blendshapes and oral-cavity geometry (teeth, gums, tongue) required to produce expressive, animation-ready 3D characters. Crucially, OmniFaceRig supports a wide gamut of topologies—humans, humanoids, long-muzzled animals (e.g., dogs), short-muzzled animals (e.g., cats, bears)—without manual intervention, landmarks, or asset-specific template setup. The pipeline output comprises up to 155 FACS blendshapes, procedurally generated inner-mouth geometry, and re-packed UVs/textures, supporting both realistic and stylized assets.
Through the public release of Omni-Bench, a dataset of 1,000 biped rigged characters with FACS blendshapes and inner-mouth geometry, OmniFaceRig enables standardized benchmarking and paves the way for further advances in large-scale, automatic facial rigging.
Pipeline Architecture
Riggability Checking
OmniFaceRig begins with an eligibility check leveraging a hybrid Vision-LLM (VLM) and classical computer vision (CV) signals. The VLM identifies semantic criteria (face type, occlusions, mouth state, teeth visibility), while CV segmentation masks validate geometric plausibility (facial region presence, eye-to-head ratio, lip curvature). A majority-vote scheme, including a landmark-quality gate for human assets, ensures robust mouth detection, mitigating silent failures common in solely landmark-driven pipelines.
Empirical results demonstrate the hybrid checker achieves >95% accuracy, recall, and F1 on a mixed validation set, outperforming both VLM-only and CV-only strategies.
Segmentation Ensemble
A four-model ensemble combines:
- Face landmark detection (high precision for human faces),
- Sapiens-based ViT parsers (pretrained and fine-tuned for humans and animals),
- Segment Anything Model 3 (SAM 3, class-agnostic for stylized assets),
- Adaptive selection for facial regions where single-model outputs fail.
This architecture achieves near-perfect face detection recall (∼99%) on Omni-Bench, with fine-tuned Sapiens dramatically improving animal face detection compared to vanilla models.
Dense Keypoint-Driven Template Registration
OmniFaceRig deploys a small library of topology-specific quad-mesh face templates: human, long-muzzle, and short-muzzle variants. Registration is performed via global rigid alignment and per-vertex non-rigid optimization, guided by dense segmentation-derived keypoints. The design intentionally omits nose anchors due to unreliable segmentation and contour regularity, especially in animals, preventing severe artifacts and pipeline crashes.
Three fallback mechanisms (mesh-surface correction, anchor-based re-projection, adaptive eye mask use) ensure robust keypoint computation and zero invalid keypoint errors.
Procedural Inner-Mouth Synthesis
Teeth, gums, and tongue are synthesized from archetype templates (human, canine, monster, flat), selected by the VLM. Placement uses ARAP deformation for initial fit and SDF-based iterative refinement to guarantee intersection-free oral-cavity geometry under expression activation. Gum and tongue construction is coupled, maintaining topology-specific adaptation across all targeted species.
Texture and UV Transfer
A repacked UV layout allocates high texel density to face and inner-mouth regions, preserving identity features and adapting appearance for generated oral components via vectorized texture transfer from the original mesh.
Blendshape Transfer and Postprocessing
Canonical FACS blendshapes are transferred using sparse triangle-wise deformation, preserving character-specific geometry. Expression-specific postprocessing (eyelid/eye-gaze handling, jaw-driven lower teeth motion, collision-aware refinement) ensures physically consistent results across diverse characters. Rig output is configurable with 13, 46, or 155 blendshapes, suitable for varying fidelity requirements.
Quantitative Results
OmniFaceRig outperforms state-of-the-art baselines (Deformation Transfer, Neural Face Rigging, RigAnyFace) in mean absolute error, worst-case alignment, and penetration rate:
| Method |
MAE (mm) |
Q95 (mm) |
Penet. (%) |
Success (%) |
| Deformation Transfer |
2.93 |
8.41 |
-- |
82.4 |
| Neural Face Rigging |
2.77 |
7.21 |
-- |
85.1 |
| RigAnyFace |
1.01 |
2.94 |
0.17 |
-- |
| OmniFaceRig |
0.85 |
2.50 |
0.05 |
99.0 |
On 300 human/humanoid Omni-Bench assets, OmniFaceRig maintains sub-millimeter error (MAE: 0.92mm, Q95: 2.71mm), negligible penetration (0.08%), and near-perfect rigging success (99.0%), with zero crashes on animal assets due to its robust keypoint computation and template design.
Qualitative and Ablation Analyses
Qualitative grids underscore cross-topology generalization, intersection-free oral-cavity geometry, and identity preservation. Ablations highlight that:
- Fine-tuned Sapiens parsers and ensemble selection are critical for animal face detection (face recall: 85–89% vs. 12–42% for baseline models; ensemble: ∼99%),
- Nose-anchor inclusion in templates causes severe deformations and pipeline failures in animals (distortions: 78%, crashes: 10%),
- Hybrid VLM+CV riggability checking is superior to individual components.
The full pipeline executes in 20–30s per asset on A100 GPUs, a 2–3 order-of-magnitude reduction over manual workflows.
Dataset Contribution: Omni-Bench
Omni-Bench provides 1,000 rigged biped 3D characters (500 humans/humanoids, 500 animals) with up to 155 auto-generated FACS blendshapes and inner-mouth geometry. Assets span realistic occupations, stylized genres, ten breeds each of cats and dogs, and additional animals (bears, tigers, etc.), all in T-pose. Each asset includes provenance—text description, 2D reference, 3D mesh—enabling multimodal generative and rigging research. Omni-Bench fills critical gaps in prior datasets, such as FACS and oral-cavity geometry for non-human assets.
Theoretical and Practical Implications
OmniFaceRig demonstrates fully automatic, robust, and generalizable facial rigging for both human and non-human characters, directly addressing the scalability limitations and anatomical biases of previous parametric and neural rigging methods. Its procedural approach to inner-mouth synthesis, combined with topology-specific template registration and multi-model segmentation, represents a practical paradigm for lifting generated assets to animation-ready rigs.
Practically, the method enables batch rigging of generated 3D characters with minimal preparation and maximal fidelity, promoting integration with text-to-3D pipelines. Theoretically, the separation of perception and geometry stages, absence of manual correspondences, and adaptive template selection pose promising directions for general-purpose mesh parameterization, transferability across anatomical domains, and zero-shot rigging.
Future avenues may involve expansion to further topology families (birds, fish, insects), adaptive template/family generation, deeper integration with neural avatar models, and online interactive rigging.
Conclusion
OmniFaceRig sets a new standard for fully automatic, inner-mouth-aware facial rigging across diverse 3D character topologies, combining robust perception, procedural geometry synthesis, and blendshape transfer. The public release of the Omni-Bench dataset provides a vital resource for benchmarking and further research. While current limitations include restricted template families and offline operation, the system architecture offers a foundation for future extension to broader anatomical and stylized spaces, strengthening the bridge between 3D asset generation and animation-ready rigging.
