T2Bs: Text-to-Character 3D Blendshape Models

• T2Bs is a framework for generating animatable 3D blendshape models from text by integrating static text-to-3D generation with dynamic video diffusion.
• It refines multi-view video outputs into registered 3D deformations using view-conditioned deformable Gaussian splatting and PCA-based extraction.
• The approach addresses gaps in existing methods by ensuring temporal consistency and detailed expression diversity for effective animation retargeting.

T2Bs, short for “Text-to-Character Blendshapes via Video Generation,” is a framework for generating animatable 3D character head morphable models from text by combining static text-to-3D generation with video diffusion (Luo et al., 12 Sep 2025). Its stated objective is not merely to synthesize a detailed static asset or a single dynamic sequence, but to produce a reusable blendshape or morphable model with registered 3D deformations, diverse facial motions, and downstream suitability for animation and retargeting. The framework is positioned around a specific gap in the literature: text-to-3D systems provide high-quality static geometry but no motion, while video and recent 4D generation methods provide motion but suffer from temporal and multi-view geometric inconsistencies (Luo et al., 12 Sep 2025).

1. Problem setting and conceptual scope

T2Bs targets the problem of generating animatable 3D character head models from text alone, including non-human and stylized characters, with enough expression diversity to support downstream animation and retargeting (Luo et al., 12 Sep 2025). This target differs from two adjacent tasks. First, it differs from static text-to-3D generation, which provides identity, appearance, and geometry but no expression space or temporal dynamics. Second, it differs from single-animation 4D generation, which may produce one moving sequence but not a controllable morphable model with registered geometry across many expressions (Luo et al., 12 Sep 2025).

The paper makes this distinction central. A morphable model requires many expressions, registered geometry across sequences, and a compact deformation basis. This is especially relevant for animal or stylized characters, where standard human face models such as FLAME are not directly applicable (Luo et al., 12 Sep 2025). The framework therefore aims to convert generated videos into coherent, registered 3D deformations that can be aggregated into a reusable blendshape model rather than treated as one-off animations.

The method also departs from optimization-heavy 4D approaches that rely on score distillation sampling (SDS) or from systems trained on limited 4D data. The paper identifies multi-view geometric inconsistency, appearance inconsistency across views and time, blur in difficult regions such as the mouth and eyes, and temporal inconsistency for occluded or reappearing regions as the principal obstacles to direct 3D reconstruction from video diffusion outputs (Luo et al., 12 Sep 2025).

2. Pipeline architecture

The T2Bs pipeline is organized as a sequence that starts from a static character asset and ends with a PCA-derived expression basis. The paper describes the core strategy as: generate a high-quality static 3D character head from text; use video diffusion to generate many expression videos of that character; convert those videos into multi-view videos; reconstruct temporally varying 3D geometry using a deformable Gaussian splatting model anchored to the static asset; extract registered meshes; and apply PCA to the per-frame geometry across all sequences (Luo et al., 12 Sep 2025).

Stage	Mechanism	Role
Static asset generation	TRELLIS3D	Provides textured 3D mesh
Motion synthesis	Image-to-video diffusion	Produces fixed-view expression video
Multi-view expansion	SV4D or 4Real-Video	Produces synchronized multi-view videos
Reconstruction	VCDGS	Recovers registered 3D deformations
Model construction	PCA	Builds orthogonal blendshapes

The pipeline begins with an off-the-shelf text-to-3D generator, specifically TRELLIS3D, to obtain a textured 3D mesh that serves as the canonical geometry, topology, and texture prior (Luo et al., 12 Sep 2025). The paper emphasizes that this prior is structurally important because later video-generated motions are noisy and inconsistent.

To obtain expression diversity, the framework uses around 20 predefined expression prompts, including examples such as blinking eyes, frowning, opening mouth, talking, screaming, smiling, tongue out, shaking head, happy, sad, and angry (Luo et al., 12 Sep 2025). Each video prompt combines the original character description with an expression prompt and the camera instruction “static camera, the camera is holding still” (Luo et al., 12 Sep 2025). A frontal rendering of the static character is then passed to an image-to-video model, yielding a fixed-view monocular sequence

$$[\mathcal{I}_{0,0}, \mathcal{I}_{0,1}, \cdots, \mathcal{I}_{0,T}],$$

where $v=0$ denotes the frontal or canonical view and $t$ indexes time (Luo et al., 12 Sep 2025).

A second sequence is produced by rendering the static 3D mesh while moving the camera in a circular trajectory, giving a fixed-time multi-view video

$$[\mathcal{I}_{0,0}, \mathcal{I}_{1,0}, \cdots, \mathcal{I}_{V,0}].$$

A 4D video diffusion model then synthesizes the full synchronized space-time view grid $\mathcal{I}_{v,t}$ for all viewpoints $v$ and times $t$ (Luo et al., 12 Sep 2025). The paper reports experiments with SV4D and 4Real-Video, and states that 4Real-Video gives better mouth/eye quality (Luo et al., 12 Sep 2025).

For each expression prompt, the system generates 3 sets of multi-view videos and manually filters out unnatural or implausible outputs (Luo et al., 12 Sep 2025). The static 3D asset is rendered at $512 \times 512$ and cropped as required by the video models (Luo et al., 12 Sep 2025).

3. View-Conditioned Deformable Gaussian Splatting

The central reconstruction module is View-Conditioned Deformable Gaussian Splatting (VCDGS), introduced to fit generated multi-view videos while tolerating their geometric and appearance inconsistencies (Luo et al., 12 Sep 2025). In the canonical pose, the character is represented as a set of static 3D Gaussian splats, each storing

• position $\mathbf{x} \in \mathbb{R}^3$,
• orientation $\mathbf{q} \in SO(3)$,
• scale $v=0$0,
• RGB color $v=0$1.

The implementation omits higher-order spherical harmonics and initializes opacity to 1 (Luo et al., 12 Sep 2025). The canonical Gaussians are initialized from the static mesh by cloning mesh vertex positions, assigning corresponding colors, setting opacity to 1, and optimizing scale and orientation by rendering loss (Luo et al., 12 Sep 2025).

Temporal deformation is modeled with an LBS-style formulation: $v=0$2 Here $v=0$3 is the canonical Gaussian position, $v=0$4 is the deformed position at time $v=0$5, $v=0$6 are fixed blending weights, and $v=0$7 are rigid transforms associated with control points (Luo et al., 12 Sep 2025). The control-point design is explicit: the appendix reports 2000 uniformly distributed control points, each Gaussian influenced by its 10 nearest control points, with weights given by normalized inverse Mahalanobis distances (Luo et al., 12 Sep 2025). The paper further states that 2000 control points outperform 200, 500, and 1000 for subtle motions such as eyelids and tongue (Luo et al., 12 Sep 2025).

The reconstruction objective compares rendered frames $v=0$8 with generated frames $v=0$9: $t$0 The paper interprets Huber as a robust pixelwise image loss and LPIPS as a perceptual feature loss (Luo et al., 12 Sep 2025).

Because the generated multi-view videos are not perfectly consistent, T2Bs explicitly makes deformation view dependent during training: $t$1 where $t$2 is the control-point position and the MLP predicts an $t$3 transform (Luo et al., 12 Sep 2025). The paper’s key rationale is that if inconsistent training views are forced to share a single geometry, severe artifacts result; view conditioning absorbs those inconsistencies while preserving the frontal view as the trusted canonical geometry. At inference time, the framework uses $t$4 as the canonical view and retains only $t$5 (Luo et al., 12 Sep 2025).

To stabilize optimization, the deformation is decomposed into a global motion term and a local residual: $t$6 The training schedule first fits $t$7, then jointly optimizes global and local deformation (Luo et al., 12 Sep 2025). A separate RefineMLP predicts appearance-related offsets

$t$8

which are used only during training to absorb color, scale, orientation, and illumination inconsistencies that would otherwise be spuriously encoded as geometry (Luo et al., 12 Sep 2025).

The paper states that VCDGS optimization uses 60,000 iterations per set of multi-view videos, and that the entire process, including multi-view generation and VCDGS fitting, takes about 1 hour on one NVIDIA A100 GPU (Luo et al., 12 Sep 2025).

4. Registration, blendshape construction, and retargeting

A defining feature of T2Bs is that reconstructed meshes remain registered because every sequence is modeled as a deformation of the same canonical geometry (Luo et al., 12 Sep 2025). Gaussian splats are associated with mesh vertices, per-frame meshes are extracted after fitting VCDGS, and the final geometry is defined relative to the canonical static asset. This registration property makes the recovered sequences directly suitable for blendshape learning.

The paper states that it applies principal component analysis (PCA) to the per-frame geometry of all video sequences to construct a set of orthogonal blendshapes (Luo et al., 12 Sep 2025). In experiments, the resulting morphable model uses 100 blendshapes (Luo et al., 12 Sep 2025). For held-out captures, the learned blendshape model is fit by minimizing

$t$9

where $$[\mathcal{I}_{0,0}, \mathcal{I}_{1,0}, \cdots, \mathcal{I}_{V,0}].$$0 is point-to-point Euclidean distance and $$[\mathcal{I}_{0,0}, \mathcal{I}_{1,0}, \cdots, \mathcal{I}_{V,0}].$$1 is pixel-wise color distance after rendering from the same camera (Luo et al., 12 Sep 2025).

Retargeting is designed for virtual characters rather than standard human faces. The paper explicitly notes that standard human landmark detectors are unsuitable for non-human characters (Luo et al., 12 Sep 2025). To build character-specific landmarks, it renders the static asset frontally and uses SAM2 + GroundingDINO to identify the left eye, right eye, and mouth, then selects landmarks on segmentation boundaries at predefined angles and maps them back to 3D using shadow mapping (Luo et al., 12 Sep 2025). The result is a set of 20 canonical landmarks covering the eyes and mouth (Luo et al., 12 Sep 2025).

Human-to-character motion transfer uses a subset of human facial landmarks: 6 landmarks per eye and 8 landmarks on the mouth (Luo et al., 12 Sep 2025). Blendshape weights are optimized with the objective

$$[\mathcal{I}_{0,0}, \mathcal{I}_{1,0}, \cdots, \mathcal{I}_{V,0}].$$2

with additional regularization around the eyes (Luo et al., 12 Sep 2025). The paper presents this stage as a 3D-aware retargeting procedure that handles exaggerated non-human facial structure better than 2D portrait-animation baselines.

5. Evaluation, baselines, and ablations

The experimental comparison includes DreamGaussian4D (DG4D), SV4D, 4Real-Video, and L4GM (Luo et al., 12 Sep 2025). The paper reports the following qualitative pattern: DG4D produces distorted geometry even when initialized with the same static Gaussian splats; SV4D produces blur, especially on complex structures and detailed facial parts; 4Real-Video is better than SV4D but still exhibits geometric distortion and inconsistency; and T2Bs refines these outputs into more stable reconstructions while additionally producing high-quality meshes (Luo et al., 12 Sep 2025). Qualitatively, the method is reported to perform best on distant novel views, eye and mouth details, complex shapes such as antlers, and preservation of plausible geometry during animation (Luo et al., 12 Sep 2025).

The user study comprises 422 samples, each rated by 10 participants, for a total of 4,220 ratings (Luo et al., 12 Sep 2025). The reported preference distributions are:

• Appearance: DG4D 0.0, SV4D 0.2, 4Real-Video 13.0, T2Bs 86.8
• Motion: DG4D 0.1, SV4D 0.4, 4Real-Video 40.9, T2Bs 58.7
• Geometry: DG4D 0.0, SV4D 0.1, 4Real-Video 15.8, T2Bs 84.2
• Overall: DG4D 0.1, SV4D 0.1, 4Real-Video 15.3, T2Bs 84.6 (Luo et al., 12 Sep 2025)

The paper also reports pairwise GPT-4o / GPTEval3D Elo ratings across three criteria. For Text-Asset Alignment, the scores are SV4D 818.5, L4GM 729.9, DG4D 1136.6, 4Real-Video 1075.2, and T2Bs 1301.5. For 3D Plausibility, they are 799.4, 734.1, 1138.3, 1126.7, and 1306.2, respectively. For Texture Details, they are 664.1, 624.5, 1157.4, 1151.4, and 1436.4 (Luo et al., 12 Sep 2025).

For blendshape expressiveness, the appendix reports that fitting the 100-blendshape model to held-out captures yields an average pixel-wise $$[\mathcal{I}_{0,0}, \mathcal{I}_{1,0}, \cdots, \mathcal{I}_{V,0}].$$3 fitting error of 0.0009 and an average 3D point-to-point error of 0.0017 relative to bounding box size (Luo et al., 12 Sep 2025). In retargeting, the paper compares against LivePortrait and states that LivePortrait struggles with exaggerated non-human eyes and mouths, while T2Bs produces more expressive and plausible retargeting because it uses coherent 3D geometry and a character-specific expression basis (Luo et al., 12 Sep 2025).

The ablation studies isolate the components of VCDGS. Without view conditioning, geometry artifacts appear and the model fails to reconcile inconsistent multi-view frames (Luo et al., 12 Sep 2025). Using SV4D rather than 4Real-Video as the multi-view source still allows T2Bs to improve over the raw source videos. The appendix reports average LPIPS/FID values of 0.1543 / 167.3 for SV4D, 0.1824 / 151.7 for 4Real-Video, 0.0880 / 59.6 for T2Bs, and 0.0882 / 56.6 for T2Bs (SV4D source) (Luo et al., 12 Sep 2025). Removing RefineMLP worsens geometry, with appendix averages changing from p2p 0.0576, NC 0.1611 for T2Bs to p2p 0.0613, NC 0.1851 without RefineMLP (Luo et al., 12 Sep 2025). Replacing LBS with direct per-Gaussian deformation also degrades smoothness and geometry, changing the appendix averages from p2p 0.0576, NC 0.1611 to p2p 0.0696, NC 0.2654 (Luo et al., 12 Sep 2025).

6. Limitations and research position

The paper identifies four principal limitations. First, T2Bs is head-only: it is designed for head animation rather than full-body articulated motion, and the paper states that blendshapes are not suitable for strongly articulated full-body non-linear motion (Luo et al., 12 Sep 2025). Second, it depends on the initial 3D asset having sufficient mouth interior geometry; reconstruction is sensitive to rapid topological changes such as mouth opening when the canonical asset does not cover interior regions well (Luo et al., 12 Sep 2025). Third, the generated videos may still contain unnatural deformations, temporal inconsistencies, or prompt failures for emotional states (Luo et al., 12 Sep 2025). Fourth, the method remains sensitive to topological change, including structures such as protruding tongues, unless such geometry is already supported by the initial asset (Luo et al., 12 Sep 2025).

Within the broader research landscape, T2Bs is presented as a route from text → character identity → diverse generated expressions → coherent registered 3D captures → animatable blendshape model (Luo et al., 12 Sep 2025). The framework builds its morphable model solely from generated data, without relying on a pre-trained human face morphable model or real facial motion-capture supervision (Luo et al., 12 Sep 2025). This suggests a specific research position: T2Bs is not primarily a new text-to-3D model, and not merely a 4D video generator, but a system for converting synthetic visual supervision into a reusable character-specific facial deformation space.