OmniFaceRig: Automated 3D Facial Rigging

• OmniFaceRig is an automatic facial rigging pipeline that converts static 3D meshes into animation-ready rigs by adding inner-mouth geometry and up to 155 blendshapes.
• It employs a two-stage process involving face template fitting and blendshape rig construction with hybrid vision-language screening and multi-model segmentation.
• The system minimizes collision issues and adapts topology robustly across diverse human and non-human assets, enhancing production efficiency.

Searching arXiv for OmniFaceRig and closely related rigging papers to ground the article in current literature. OmniFaceRig is a fully automatic facial rigging pipeline that converts a static, surface-only 3D character mesh into an animation-ready, inner-mouth-aware FACS rig with up to 155 blendshapes, procedurally fitted teeth, gums, and tongue, and re-packed UV/texture. It is designed for generated 3D assets that otherwise lack facial controls, deformation-friendly topology, and oral-cavity geometry, and it targets diverse topologies including humans, humanoids, long-muzzled animals, and short-muzzled animals, without manual landmarks, user-provided templates, or per-asset setup (Wang et al., 6 Jun 2026).

1. Problem setting and production context

Facial rigging remains a major bottleneck in 3D character production because attractive static meshes are not equivalent to production-ready facial assets. OmniFaceRig addresses the gap between a generated outer surface and an animation-ready character: the input mesh typically lacks facial control, correct deformation topology, and the oral-cavity geometry needed for convincing open-mouth expressions. The method therefore treats rig generation as a full asset-construction problem rather than as blendshape transfer alone (Wang et al., 6 Jun 2026).

A central emphasis is inner-mouth completion. The pipeline explicitly models teeth, gums, and tongue because open-mouth expressions expose the interior, and a rig without correctly placed oral structures can appear hollow or deform incorrectly. The paper states that existing pipelines often ignore this, while manual placement of oral structures remains especially burdensome for stylized and non-human characters. OmniFaceRig is accordingly framed as an unattended path from static generated characters to animation-ready facial rigs across both human and non-human topologies (Wang et al., 6 Jun 2026).

A common misunderstanding is to interpret OmniFaceRig as a facial animation or motion-generation system. Its actual scope is earlier in the pipeline: it transforms a static mesh into a riggable asset. This distinction matters because the technical requirements are dominated by topology adaptation, registration, and procedural geometry synthesis rather than by temporal modeling.

2. Two-stage pipeline and automatic riggability screening

OmniFaceRig is organized as two stages. Stage 1, “Face Template Fitting,” localizes the face, parses it, extracts keypoints, and fits a canonical template mesh to the input geometry. Stage 2, “Blendshape Rig Construction,” fuses the fitted face back into the original asset, synthesizes and places inner-mouth geometry, transfers FACS blendshapes, and rebuilds UVs/textures (Wang et al., 6 Jun 2026).

The first step is an automatic riggability check. The system uses a hybrid VLM + CV design because no single signal is reliable enough across humans, humanoids, and animals. A frontal render is passed to a vision-LLM for semantic judgments such as character type, occlusion state, and whether the mouth state is suitable for rigging. In parallel, a multi-model segmentation ensemble checks for the presence and plausibility of face, eye, and mouth regions. For human-like assets, mouth landmarks are also used after a quality gate based on lip evenness; if the landmarks are too irregular, they are treated as unreliable. The final eligibility decision is a conjunction of these signals, with mouth presence handled by majority vote for human-like characters. On a roughly 500-asset validation set, the paper reports over 95% accuracy/recall/F1 for this hybrid checker (Wang et al., 6 Jun 2026).

Face parsing is then handled by an ensemble built around Sapiens-1B, together with a face landmark detector and SAM 3. The fusion is region-adaptive rather than a simple vote. Human faces rely on landmark detection and the human parser; stylized humanoids use a fine-tuned stylized parser; animals fall back to a humanoid parser plus SAM 3 where needed. The resulting masks are converted into dense 2D and 3D keypoints by reading mask contours and projecting them back onto the mesh via ray casting. The paper attributes nearly complete face detection coverage on Omni-Bench to this adaptive, region-level fusion, reporting about 99% overall recall for the full ensemble (Wang et al., 6 Jun 2026).

This architecture reflects a specific design choice: failure should occur early and explicitly, at screening time, rather than later through unstable deformation. A plausible implication is that the riggability checker is not merely a preprocessing convenience but a structural safeguard against unsupported topology families and geometrically invalid assets.

3. Template registration, nose-landmark-free fitting, and topology adaptation

Template registration is performed with three topology-specific face templates: human, long-muzzle, and short-muzzle. Long-muzzle targets animals such as dogs, wolves, and foxes; short-muzzle targets cats, bears, rabbits, tigers, and similar species with compact snouts. For non-human characters, the system also selects topology-specific inner-mouth templates, again without exposing users to category-specific tuning (Wang et al., 6 Jun 2026).

A key design decision is that the template fit is “nose-landmark-free.” The paper argues that nose segmentation is too unstable across animals and stylized characters, and that even detected nose contours are too irregular to provide reliable uniformly sampled anchors. OmniFaceRig therefore excludes nose anchors entirely and allows the optimization to infer the nose region from surrounding anchors. The paper identifies this decision as one reason the system generalizes across species without category-specific tuning (Wang et al., 6 Jun 2026).

The fitting procedure combines rigid and nonrigid registration. First, the template is globally aligned by solving for scale, rotation, and translation to minimize squared keypoint error. It is then deformed under a composite objective with robust keypoint fitting, smooth deformation, edge-length preservation, triangle-shape preservation, flipped-triangle prevention, and regularization. The paper states that this stage yields near-zero alignment error on normalized meshes, and that the nose-free design plus the flip penalty eliminates crashes on the animal assets in Omni-Bench (Wang et al., 6 Jun 2026).

Several fallback mechanisms make keypoint extraction stable. Border keypoints that land outside the mesh are snapped back to the nearest mask boundary; when 2D-to-3D projection fails, the system searches the nearest mesh-surface pixel; if that also fails, the point is estimated from neighboring resolved keypoints. Eye treatment is also adaptive: humans use landmark-based eyes, while non-humans use segmentation-based eyes to avoid overly large convex hulls (Wang et al., 6 Jun 2026).

After fitting, the canonical face is fused into the original asset by cutting out the old face region and welding in the new face topology, including radial edge flow around the eyes and mouth. This step is significant because OmniFaceRig is not limited to pose-matching an existing surface; it explicitly replaces local topology with animation-friendly topology before blendshape transfer.

4. Inner-mouth construction, collision-aware fitting, and blendshape rig construction

The second stage constructs the production rig. OmniFaceRig synthesizes the oral cavity from a small set of inner-mouth archetypes: human, canine, monster, and flat. The VLM chooses the archetype during Stage 1. Teeth geometry is warped into place using RBF deformation and then refined with an ARAP + SDF procedure. The ARAP step provides an initial rigidly consistent fit, while the SDF-based refinement pushes intersecting geometry outward to reduce teeth-face penetration. Gum and tongue geometry are fitted as well (Wang et al., 6 Jun 2026).

The paper emphasizes collision-aware placement because inner-mouth usability is tested under strong expressions rather than in the neutral pose alone. For non-human characters, collision-aware inner-mouth fitting is used specifically to reduce teeth-face intersections without requiring category-specific manual tuning. This is one of the method’s main departures from pipelines that transfer only external facial blendshapes (Wang et al., 6 Jun 2026).

Once geometry insertion is complete, UVs are rebuilt and textures are re-baked. The face receives much higher texel density than the rest of the mesh, and the inner mouth is placed on its own texture channel. FACS blendshapes are then transferred from a canonical blendshape template to the merged asset using sparse deformation transfer, which preserves the asset’s identity while adapting expressions to the new topology. Expression-specific post-processing follows, including eyelid contact correction, eye-gaze rotation around virtual eyeballs, jaw-linked lower teeth motion, and repeated SDF-based collision checks for problematic shapes (Wang et al., 6 Jun 2026).

The pipeline supports three output tiers: Core with 13 shapes, Additional with 46, and Full with 155. This tiered design suggests multiple deployment regimes, from lighter-weight interactive use to fuller production assets, while remaining within a common FACS-style rigging framework.

5. Omni-Bench and empirical performance

Omni-Bench is the public benchmark dataset released with OmniFaceRig. It contains 1,000 biped 3D characters with FACS facial blendshapes and inner-mouth geometry, all processed by OmniFaceRig and released in T-pose. The split is 500 humans/humanoids and 500 animals. The human side includes realistic humans across 13 occupations plus stylized humanoid characters. The animal side includes 150 cats from 10 breeds, 150 dogs from 10 breeds, and 200 additional animal characters such as bears, tigers, lions, foxes, wolves, rabbits, and deer. Each asset includes provenance data comprising a text prompt, an intermediate 2D reference image, and the final generated 3D mesh. The paper presents Omni-Bench as the first dataset of this kind to combine full-character assets, FACS blendshapes, and inner-mouth geometry across human and animal topology families (Wang et al., 6 Jun 2026).

Quantitatively, OmniFaceRig is evaluated for registration quality, collision behavior, rigging success, and parsing recall. On 200 human/humanoid heads, it achieves mean vertex error of 0.85 mm and 95th percentile error of 2.50 mm, compared with 1.01 mm and 2.94 mm for RigAnyFace. Its penetration rate for inner-mouth collisions is 0.05%, compared with 0.17% for RigAnyFace, described in the paper as a 3.4× reduction. On a 300-asset human/humanoid subset of Omni-Bench, the success rate is 99.0%, whereas deformation-transfer and NFR baselines are notably lower. The full pipeline runs in about 20–30 seconds per asset on a single A100 GPU, including I/O (Wang et al., 6 Jun 2026).

The segmentation ablation explains why the ensemble is necessary. Face landmark detection is strong on humans but poor on animals; standard Sapiens is human-biased; SAM 3 is broad but coarse; and the fine-tuned parser improves animal coverage dramatically. Together, the four-model ensemble reaches about 99% overall face detection recall. This result is methodologically important because template fitting quality depends directly on face localization coverage across stylized and non-human categories (Wang et al., 6 Jun 2026).

These results support a narrow but consequential claim: the pipeline is reliable after screening on supported inputs. They do not imply universal riggability for arbitrary 3D creatures.

6. Relation to neighboring research areas

OmniFaceRig occupies a different technical niche from systems that generate facial motion, edit face images, or retarget expressions across already-rigged meshes. Ex-Omni, for example, augments omni-modal LLMs with speech-accompanied 3D facial animation and predicts ARKit-52 blendshape coefficients as a temporally aligned control signal; OmniFaceRig instead begins from a static surface-only 3D mesh and constructs the rig itself, including inner-mouth geometry and UV/texture restructuring (Zhang et al., 6 Feb 2026). This suggests a complementary relationship: OmniFaceRig can supply the animation-ready asset, while Ex-Omni-style methods can supply the time-varying control parameters.

RigFace addresses controllable face editing in portrait images by disentangling identity, background, pose, expression, and lighting within a Stable Diffusion framework, using crude 3D face models as control signals. OmniFaceRig operates on a different representation entirely: it edits neither portraits nor latent image features, but rather topology, oral geometry, and FACS blendshape structure in 3D character meshes (Wei et al., 4 Feb 2025).

Neural Face Rigging (NFR) similarly targets automatic rigging and retargeting for arbitrary facial meshes, with human-interpretable controls and triangulation-agnostic deformation. OmniFaceRig differs in its emphasis on fully automatic asset preparation from surface-only meshes, topology-family selection, and explicit procedural construction of teeth, gums, and tongue. NFR is primarily a deformation model over facial meshes; OmniFaceRig is a full asset-construction pipeline (Qin et al., 2023).

A common misconception is that “automatic rigging” refers only to transferring deformation bases. OmniFaceRig makes a broader claim: animation readiness depends on screening, parsing, registration, topology replacement, oral-cavity synthesis, collision handling, UV repacking, and post-processed blendshape transfer as a single pipeline. That broader scope is the basis for its relevance to generated humans, humanoids, and animals.

7. Limitations, failure regimes, and scope of generalization

The method is designed for visible-faced characters whose topology fits within the three template families. The authors state that it can fail or reject assets such as birds with sharp beaks, fish-like faces, insects with oversized compound eyes, or characters with extremely unusual snout structures that do not map to human, long-muzzle, or short-muzzle templates. In practice, such assets may be screened out by the riggability checker, or later fitting stages may produce unreliable fits (Wang et al., 6 Jun 2026).

The pipeline is also an offline batch process rather than a real-time interactive tool. Its automation therefore reduces designer effort but does not turn rigging into an interactive editing system. Likewise, “fully automatic” should not be read as “universally applicable”: the automation is bounded by the template families, by the requirement that the face be visible and semantically parseable, and by the assumption that an inserted canonical face topology can be welded into the source asset (Wang et al., 6 Jun 2026).

Within those bounds, OmniFaceRig represents a specific synthesis of semantic screening, multi-model parsing, dense keypoint-driven registration, topology-specific template fitting, procedural oral-cavity synthesis, and collision-aware blendshape transfer. A plausible implication is that its main contribution is not a single algorithmic primitive but the closure of multiple long-standing gaps in one unattended pipeline: face eligibility, cross-species template fitting, inner-mouth generation, and asset-level delivery of FACS rigs.