ScaffoldAvatar: Fine-Grained 3D Face Synthesis

• ScaffoldAvatar is a method for generating high-fidelity 3D head avatars using patch-based local expression modeling coupled with Gaussian splatting.
• It divides the face into 432 overlapping patches, yielding 8208 localized expression parameters for precise regional control and detailed close-up realism.
• The approach integrates view-conditioned color MLPs, color-based densification, and progressive high-resolution training to enhance sharpness and motion fidelity.

Searching arXiv for the specified paper and closely related context. ScaffoldAvatar is a method for generating high-fidelity real-time animated sequences of photorealistic $3$D head avatars by coupling locally defined facial expressions with $3$D Gaussian splatting. It was introduced as a capture-driven avatar model that conditions dynamics on patch-based local expression features, couples facial patches with anchor points of Scaffold-GS, and synthesizes $3$D Gaussians on-the-fly conditioned by patch expressions and viewing direction. The stated objective is close-up facial realism, including facial microfeatures, skin furrowing, and finer-scale facial movements that are not well handled by global expression spaces (Aneja et al., 14 Jul 2025).

1. Representational premise and problem setting

ScaffoldAvatar is motivated by the claim that global expression codes are too coarse to model fine facial motion and appearance. Prior avatar systems are described as conditioning deformations on a single low-dimensional expression vector from a global $3$D face model such as FLAME or BFM. According to the method description, this is sufficient for broad facial motion but struggles with localized muscle motion around the lips, eyes, and cheeks; micro-expressions such as wrinkles, furrows, and skin creases; expression-dependent appearance such as blood flow or subtle shading changes; and fine geometric retargeting required for close-up realism (Aneja et al., 14 Jul 2025).

The central alternative is a patch-based facial model with many localized coefficients. The tracked mesh is split into $P = 432$ overlapping patches, and for each patch a local blendshape model is fit from $K = 20$ static scans. This yields $P \cdot (K-1) = 432 \cdot 19 = 8208$ expression parameters, compared with about $100$ in FLAME. The paper presents this increase in representational capacity as the key reason the method can model localized deformation more accurately than global expression spaces (Aneja et al., 14 Jul 2025).

The practical consequence is region-specific control. Because each facial region has its own expression parameters, the avatar can move the mouth without over-driving cheeks or forehead, model localized wrinkles only where they occur, preserve sharp boundaries and region-specific details, and avoid the smoothing associated with a single global code. A common misunderstanding would be that ScaffoldAvatar simply replaces global control with fully independent local codes; the reported formulation instead preserves both local and global expression features, using local descriptors for fine motion while retaining a global descriptor for whole-face coherence.

2. Patch expression model and extraction pipeline

The capture pipeline begins with multi-view performance data and tracked meshes. The reported setup uses $9$ synchronized cameras in a hemisphere, with $8$ training views and $3$0 validation view, $3$1MP / $3$2MP cameras at $3$3 fps, $3$4–$3$5 performance sequences per actor, and topology-consistent tracked meshes from a high-quality tracker (Aneja et al., 14 Jul 2025).

The patch layout is defined directly on the tracked face mesh. For each patch $3$6, the mean vertex position of the vertices in the patch is computed, the closest mesh vertex is chosen as the patch center $3$7, and this center is kept fixed across the dataset. Each patch also has a local coordinate frame defined by a TBNP matrix,

$3$8

where $3$9, $3$0, and $3$1 are tangent, bitangent, and normal, and $3$2 is the patch position.

The local geometric model for a patch is written as

$3$3

where $3$4 are patch-specific blendweights. For each frame, the patch blendweights are recovered by fitting the local model to the tracked mesh $3$5 through the least-squares term

$3$6

together with the regularizers

$3$7

and

$3$8

The full fitting objective is

$3$9

with optimum

$3$0

Repeated across time, this produces a sequence of patch-wise local expression codes $3$1. This extraction stage is not a peripheral preprocessing step; it is the mechanism by which ScaffoldAvatar obtains localized expression control at the same spatial granularity used later for Gaussian synthesis.

3. Coupling facial patches with Scaffold-GS anchors

The defining architectural move in ScaffoldAvatar is to couple patch-local expression codes with a Scaffold-GS-style hierarchical Gaussian generator. Each patch center $3$2 acts as the parent for a set of scaffold anchors, and the patch transform $3$3 causes anchors to move with the corresponding facial region over time (Aneja et al., 14 Jul 2025).

For patch $3$4, the anchors are denoted

$3$5

with each anchor

$3$6

Here $3$7 is the anchor position in patch-local space, $3$8 is anchor scale, $3$9 is anchor opacity, and $P = 432$0 is a learned anchor feature. The anchor’s global position is obtained by

$P = 432$1

Expression conditioning is explicitly two-level. Local patch blendweights are mapped through a patch MLP,

$P = 432$2

while all patches are concatenated into a global expression descriptor,

$P = 432$3

The paper emphasizes that $P = 432$4 captures fine region-specific motion and $P = 432$5 preserves whole-face coherence. This dual encoding clarifies that patch-based control is not meant to fragment the face into unrelated local controllers.

Each anchor spawns $P = 432$6 Gaussians. Their parameters are predicted by small MLPs conditioned on anchor feature, local expression, global expression, and view direction. Color is predicted by a per-patch color MLP,

$P = 432$7

The final Gaussian scale is

$P = 432$8

and the Gaussian position in global space is

$P = 432$9

The resulting workflow is: patch expression to patch latent feature; anchor plus expressions plus view direction to Gaussian attributes; Gaussians spawned on-the-fly around each anchor; and rasterization of the resulting Gaussian set. In the paper’s framing, this is what enables both real-time rendering and high local detail.

4. View dependence, rasterization, and optimization strategy

Viewing direction is explicitly incorporated into the prediction of Gaussian scale, rotation, opacity, and color. ScaffoldAvatar is therefore not only expression-driven but also view-adaptive, with the stated goal of representing specularities, view-dependent shading, and facial appearance changes as the camera moves (Aneja et al., 14 Jul 2025).

The final Gaussian primitives are passed to a $K = 20$0D Gaussian splatting rasterizer,

$K = 20$1

In the appendix formulation, pixel color is accumulated by alpha compositing of overlapping Gaussians,

$K = 20$2

The paper states that a per-patch color MLP is crucial for sharp zoom-ins because different facial regions need different color dynamics; it also states that direct use of a single global color MLP leads to less crisp close-up details.

Training at $K = 20$3K resolution is supported by two reported strategies. The first is color-based densification. Instead of standard $K = 20$4DGS-style densification based on position gradients, ScaffoldAvatar uses view-space color gradients as the heuristic for adding and pruning anchors. The paper argues that color gradients better highlight regions with high-frequency visual error, including wrinkles, freckles, lip boundaries, skin pores, and shadow transitions, and reports that color-based densification converges faster, yields sharper results, and better reconstructs fine facial details.

The second strategy is progressive training. Rather than train directly at $K = 20$5K resolution, the schedule proceeds from $K = 20$6K to $K = 20$7K to $K = 20$8K. The ablation notes a typical schedule of the first $K = 20$9k iterations at $P \cdot (K-1) = 432 \cdot 19 = 8208$0K, the next $P \cdot (K-1) = 432 \cdot 19 = 8208$1k at $P \cdot (K-1) = 432 \cdot 19 = 8208$2K, and a final high-resolution stage at $P \cdot (K-1) = 432 \cdot 19 = 8208$3K. The method summary also gives the training objective as

$P \cdot (K-1) = 432 \cdot 19 = 8208$4

and describes training with RGB, SSIM, LPIPS, spatial, and scale regularization. Taken together, these design choices suggest that high-resolution supervision is only effective when optimization progressively stabilizes the representation and densification allocates capacity to visually demanding regions.

5. Reported empirical performance

The reported evaluation compares ScaffoldAvatar with GaussianAvatars, GHA, and NPGA on novel view synthesis and self-reenactment, and distinguishes a $P \cdot (K-1) = 432 \cdot 19 = 8208$5K version from the full $P \cdot (K-1) = 432 \cdot 19 = 8208$6K model (Aneja et al., 14 Jul 2025). The qualitative claims are consistent across both tasks: sharper facial zoom-ins, more natural facial motion, better wrinkles and furrows, freckles and other skin microfeatures, less blur in close-ups, and fewer artifacts around sensitive regions such as the lips.

Method	Novel View Synthesis	Self-Reenactment
GaussianAvatars	26.76 / 0.9082 / 0.1489	24.52 / 0.9078 / 0.1952
GHA	28.93 / 0.9366 / 0.1335	26.82 / 0.9395 / 0.1911
NPGA	30.15 / 0.9398 / 0.1312	27.44 / 0.9335 / 0.1857
ScaffoldAvatar $P \cdot (K-1) = 432 \cdot 19 = 8208$7	32.19 / 0.9653 / 0.1260	29.82 / 0.9515 / 0.1813
ScaffoldAvatar $P \cdot (K-1) = 432 \cdot 19 = 8208$8	34.48 / 0.9712 / 0.1259	30.37 / 0.9540 / 0.1797

The table reports PSNR / SSIM / LPIPS. For novel view synthesis, the full $P \cdot (K-1) = 432 \cdot 19 = 8208$9K model reaches PSNR $100$0, SSIM $100$1, and LPIPS $100$2, compared with $100$3, $100$4, and $100$5 for NPGA. For self-reenactment, the full model reaches $100$6, $100$7, and $100$8, compared with $100$9, $9$0, and $9$1 for NPGA. The paper’s interpretation is that patch-level expressions and high-resolution Gaussian synthesis materially improve both view synthesis fidelity and reenactment quality, especially in close-up regimes.

The comparison is also qualitative. GaussianAvatars is described as blurry and unable to model expression-dependent details well; GHA is described as improving dynamics but showing grid-like artifacts and blurriness; NPGA is described as synthesizing wrinkles but still appearing blurry on zoom-ins. ScaffoldAvatar is presented as especially strong in fine-detail reconstruction.

6. Ablations, scope, and methodological significance

The ablation study attributes the reported gains to three core ingredients: patch expressions, per-patch color modeling, and high-resolution optimization procedures. Removing patch expressions introduces artifacts around the lips and blurs details. Removing the per-patch color MLP reduces close-up sharpness. Replacing color-based densification with standard position-based densification hurts sharpness. Skipping progressive training slows convergence and reduces final quality (Aneja et al., 14 Jul 2025).

These ablations matter because they isolate where the method departs from prior Gaussian avatar pipelines. Patch expressions are not presented merely as a higher-dimensional latent code; they are tied to a spatially localized geometric model and to localized Gaussian generation. Likewise, the color pathway is not globally shared but partitioned at patch level, and densification is guided by view-space color gradients rather than only geometric heuristics. A plausible implication is that ScaffoldAvatar’s improvements arise from aligning representational units, motion descriptors, and appearance predictors at the same local facial scale.

The scope of the method is equally clear from the reported pipeline. ScaffoldAvatar begins with multi-view capture, tracked meshes, and a patch-based geometric $9$2D face model. This places it in the setting of high-fidelity performance-driven head avatars rather than text-to-avatar generation or marketplace asset retrieval. The term “scaffold” in ScaffoldAvatar refers to its use of Scaffold-GS anchors as a hierarchical scene representation coupled to patch-local facial dynamics, not to an abstract prompt scaffold.

In summary, ScaffoldAvatar defines a localized alternative to global-expression Gaussian avatars. Its stated contribution is the integration of $9$3 overlapping facial patches, $9$4 patch expression parameters, patch-attached Scaffold-GS anchors, view-conditioned on-the-fly Gaussian synthesis, per-patch color prediction, color-based densification, and progressive $9$5K-to-$9$6K training into a real-time head avatar system aimed at close-up photorealism (Aneja et al., 14 Jul 2025).