---
title: 'ScaffoldAvatar: Fine-Grained 3D Face Synthesis'
url: https://www.emergentmind.com/topics/scaffoldavatar
type: topic
---

# ScaffoldAvatar: Fine-Grained 3D Face Synthesis

Searching arXiv for the specified paper and closely related context.
ScaffoldAvatar is a method for generating high-fidelity real-time animated sequences of photorealistic \(3\)D head avatars by coupling locally defined facial expressions with \(3\)D Gaussian splatting. It was introduced as a capture-driven avatar model that conditions dynamics on patch-based local expression features, couples facial patches with anchor points of Scaffold-GS, and synthesizes \(3\)D Gaussians on-the-fly conditioned by patch expressions and viewing direction. The stated objective is close-up facial realism, including facial microfeatures, skin furrowing, and finer-scale facial movements that are not well handled by global expression spaces [2507.10542].

## 1. Representational premise and problem setting

ScaffoldAvatar is motivated by the claim that global expression codes are too coarse to model fine facial motion and appearance. Prior avatar systems are described as conditioning deformations on a single low-dimensional expression vector from a global \(3\)D face model such as FLAME or BFM. According to the method description, this is sufficient for broad facial motion but struggles with localized muscle motion around the lips, eyes, and cheeks; micro-expressions such as wrinkles, furrows, and skin creases; expression-dependent appearance such as blood flow or subtle shading changes; and fine geometric retargeting required for close-up realism [2507.10542].

The central alternative is a patch-based facial model with many localized coefficients. The tracked mesh is split into \(P = 432\) overlapping patches, and for each patch a local blendshape model is fit from \(K = 20\) static scans. This yields \(P \cdot (K-1) = 432 \cdot 19 = 8208\) expression parameters, compared with about \(100\) in FLAME. The paper presents this increase in representational capacity as the key reason the method can model localized deformation more accurately than global expression spaces [2507.10542].

The practical consequence is region-specific control. Because each facial region has its own expression parameters, the avatar can move the mouth without over-driving cheeks or forehead, model localized wrinkles only where they occur, preserve sharp boundaries and region-specific details, and avoid the smoothing associated with a single global code. A common misunderstanding would be that ScaffoldAvatar simply replaces global control with fully independent local codes; the reported formulation instead preserves both local and global expression features, using local descriptors for fine motion while retaining a global descriptor for whole-face coherence.

## 2. Patch expression model and extraction pipeline

The capture pipeline begins with multi-view performance data and tracked meshes. The reported setup uses \(9\) synchronized cameras in a hemisphere, with \(8\) training views and \(1\) validation view, \(12\)MP / \(20\)MP cameras at \(24\) fps, \(5\)–\(8\) performance sequences per actor, and topology-consistent tracked meshes from a high-quality tracker [2507.10542].

The patch layout is defined directly on the tracked face mesh. For each patch \(p\), the mean vertex position of the vertices in the patch is computed, the closest mesh vertex is chosen as the patch center \(\mathbf{v}_p\), and this center is kept fixed across the dataset. Each patch also has a local coordinate frame defined by a TBNP matrix,
$$
\mathcal{T}_p =
\begin{bmatrix}
\mathbf{t}_p^x & \mathbf{b}_p^x & \mathbf{n}_p^x & \mathbf{v}_p^x \\
\mathbf{t}_p^y & \mathbf{b}_p^y & \mathbf{n}_p^y & \mathbf{v}_p^y \\
\mathbf{t}_p^z & \mathbf{b}_p^z & \mathbf{n}_p^z & \mathbf{v}_p^z \\
0 & 0 & 0 & 1
\end{bmatrix},
$$
where \(\mathbf{t}_p\), \(\mathbf{b}_p\), and \(\mathbf{n}_p\) are tangent, bitangent, and normal, and \(\mathbf{v}_p\) is the patch position.

The local geometric model for a patch is written as
$$
\mathbf{X}_{p}^{\mathcal{S}} =
\mathbf{X}_{p}^{\mathbf{S}_0} +
\sum_{i=1}^{K-1} \beta_{p,i}
\left(
\mathbf{X}_{p}^{\mathbf{S}_i} - \mathbf{X}_{p}^{\mathbf{S}_0}
\right),
$$
where \(\beta_{p,i}\) are patch-specific blendweights. For each frame, the patch blendweights are recovered by fitting the local model to the tracked mesh \(\mathbf{X}^t\) through the least-squares term
$$
\mathcal{E}_{ls} =
\sum_{p \in \mathcal{P}}
\left\|
\mathbf{X}_p^t - R \mathbf{X}_p^{\mathcal{S}}
\right\|_2^2,
$$
together with the regularizers
$$
\mathcal{E}_{reg} =
\sum_{p \in \mathcal{P}}
\sum_{i=1}^{K-1}
\|
\beta_{p,i}
\|_2,
$$
and
$$
\mathcal{E}_{o} =
\sum_{p \in \mathcal{P}}
\sum_{q \in \mathcal{N}(p)}
\sum_{i=1}^{K-1}
\|
\beta_{p,i} - \beta_{q,i}
\|_1.
$$
The full fitting objective is
$$
\mathcal{E}_{PBS} =
\lambda_{ls}\mathcal{E}_{ls} +
\lambda_{reg}\mathcal{E}_{reg} +
\lambda_{o}\mathcal{E}_{o},
$$
with optimum
$$
\boldsymbol{\beta}^* =
\arg\min_{\boldsymbol{\beta}}
\mathcal{E}_{PBS}.
$$

Repeated across time, this produces a sequence of patch-wise local expression codes \(\{\boldsymbol{\beta}^1, \boldsymbol{\beta}^2, \ldots, \boldsymbol{\beta}^T\}\). This extraction stage is not a peripheral preprocessing step; it is the mechanism by which ScaffoldAvatar obtains localized expression control at the same spatial granularity used later for Gaussian synthesis.

## 3. Coupling facial patches with Scaffold-GS anchors

The defining architectural move in ScaffoldAvatar is to couple patch-local expression codes with a Scaffold-GS-style hierarchical Gaussian generator. Each patch center \(\mathbf{v}_p\) acts as the parent for a set of scaffold anchors, and the patch transform \(\mathcal{T}_p\) causes anchors to move with the corresponding facial region over time [2507.10542].

For patch \(p\), the anchors are denoted
$$
\mathcal{A}_p = \{\mathbf{A}_{p,1}, \mathbf{A}_{p,2}, \dots, \mathbf{A}_{p,N}\},
$$
with each anchor
$$
\mathbf{A}_{p,i} =
\{ \boldsymbol{\mu}_{p,i}, \mathbf{s}_{p,i}, \alpha_{p,i} \}
\cup
\{ \mathbf{f}_{p,i} \}.
$$
Here \(\boldsymbol{\mu}_{p,i}\) is the anchor position in patch-local space, \(\mathbf{s}_{p,i}\) is anchor scale, \(\alpha_{p,i}\) is anchor opacity, and \(\mathbf{f}_{p,i}\) is a learned anchor feature. The anchor’s global position is obtained by
$$
\boldsymbol{\mu}_{p,i}^{\text{glob}} =
\mathcal{T}_p
\begin{bmatrix}
\boldsymbol{\mu}_{p,i} \\
1
\end{bmatrix}.
$$

Expression conditioning is explicitly two-level. Local patch blendweights are mapped through a patch MLP,
$$
\mathbf{e}_p = \mathbf{P}(\boldsymbol{\beta}_p),
$$
while all patches are concatenated into a global expression descriptor,
$$
\mathbf{e}_g = \mathbf{G}(\boldsymbol{\beta}_1, \dots, \boldsymbol{\beta}_P).
$$
The paper emphasizes that \(\mathbf{e}_p\) captures fine region-specific motion and \(\mathbf{e}_g\) preserves whole-face coherence. This dual encoding clarifies that patch-based control is not meant to fragment the face into unrelated local controllers.

Each anchor spawns \(L = 5\) Gaussians. Their parameters are predicted by small MLPs conditioned on anchor feature, local expression, global expression, and view direction. Color is predicted by a per-patch color MLP,
$$
\hat{c}_{0:L-1} = \mathcal{F}_{c}^{p}(\mathbf{f}_p;\mathbf{e}_p;\mathbf{e}_g;d_v).
$$
The final Gaussian scale is
$$
s_g = \hat{s}\, s,
$$
and the Gaussian position in global space is
$$
\boldsymbol{\mu}_g =
\boldsymbol{\mu}^{\text{glob}} + s\,\mathbf{R}\,\hat{\boldsymbol{\mu}}.
$$

The resulting workflow is: patch expression to patch latent feature; anchor plus expressions plus view direction to Gaussian attributes; Gaussians spawned on-the-fly around each anchor; and rasterization of the resulting Gaussian set. In the paper’s framing, this is what enables both real-time rendering and high local detail.

## 4. View dependence, rasterization, and optimization strategy

Viewing direction is explicitly incorporated into the prediction of Gaussian scale, rotation, opacity, and color. ScaffoldAvatar is therefore not only expression-driven but also view-adaptive, with the stated goal of representing specularities, view-dependent shading, and facial appearance changes as the camera moves [2507.10542].

The final Gaussian primitives are passed to a \(3\)D Gaussian splatting rasterizer,
$$
I_{\text{RGB}} =
\mathcal{R}(\boldsymbol{\mu}_g; \mathbf{s}_g; \mathbf{q}; \alpha_g; \mathbf{c}_g).
$$
In the appendix formulation, pixel color is accumulated by alpha compositing of overlapping Gaussians,
$$
\boldsymbol{C} =
\sum_{i=1}^{N}
\boldsymbol{c}_i \alpha_i
\prod_{j=1}^{i-1}(1-\alpha_j).
$$
The paper states that a per-patch color MLP is crucial for sharp zoom-ins because different facial regions need different color dynamics; it also states that direct use of a single global color MLP leads to less crisp close-up details.

Training at \(3\)K resolution is supported by two reported strategies. The first is color-based densification. Instead of standard \(3\)DGS-style densification based on position gradients, ScaffoldAvatar uses view-space color gradients as the heuristic for adding and pruning anchors. The paper argues that color gradients better highlight regions with high-frequency visual error, including wrinkles, freckles, lip boundaries, skin pores, and shadow transitions, and reports that color-based densification converges faster, yields sharper results, and better reconstructs fine facial details.

The second strategy is progressive training. Rather than train directly at \(3\)K resolution, the schedule proceeds from \(1\)K to \(2\)K to \(3\)K. The ablation notes a typical schedule of the first \(20\)k iterations at \(1\)K, the next \(35\)k at \(2\)K, and a final high-resolution stage at \(3\)K. The method summary also gives the training objective as
$$
\mathcal{L} =
\mathcal{L}_{rgb} +
\lambda_{patch}\mathcal{L}_{patch} +
\lambda_{xyz}\mathcal{L}_{xyz} +
\lambda_{scale}\mathcal{L}_{scale},
$$
and describes training with RGB, SSIM, LPIPS, spatial, and scale regularization. Taken together, these design choices suggest that high-resolution supervision is only effective when optimization progressively stabilizes the representation and densification allocates capacity to visually demanding regions.

## 5. Reported empirical performance

The reported evaluation compares ScaffoldAvatar with GaussianAvatars, GHA, and NPGA on novel view synthesis and self-reenactment, and distinguishes a \(1\)K version from the full \(3\)K model [2507.10542]. The qualitative claims are consistent across both tasks: sharper facial zoom-ins, more natural facial motion, better wrinkles and furrows, freckles and other skin microfeatures, less blur in close-ups, and fewer artifacts around sensitive regions such as the lips.

| Method | Novel View Synthesis | Self-Reenactment |
|---|---|---|
| GaussianAvatars | 26.76 / 0.9082 / 0.1489 | 24.52 / 0.9078 / 0.1952 |
| GHA | 28.93 / 0.9366 / 0.1335 | 26.82 / 0.9395 / 0.1911 |
| NPGA | 30.15 / 0.9398 / 0.1312 | 27.44 / 0.9335 / 0.1857 |
| ScaffoldAvatar \(1\text{K}\) | 32.19 / 0.9653 / 0.1260 | 29.82 / 0.9515 / 0.1813 |
| ScaffoldAvatar \(3\text{K}\) | 34.48 / 0.9712 / 0.1259 | 30.37 / 0.9540 / 0.1797 |

The table reports PSNR / SSIM / LPIPS. For novel view synthesis, the full \(3\)K model reaches PSNR \(34.48\), SSIM \(0.9712\), and LPIPS \(0.1259\), compared with \(30.15\), \(0.9398\), and \(0.1312\) for NPGA. For self-reenactment, the full model reaches \(30.37\), \(0.9540\), and \(0.1797\), compared with \(27.44\), \(0.9335\), and \(0.1857\) for NPGA. The paper’s interpretation is that patch-level expressions and high-resolution Gaussian synthesis materially improve both view synthesis fidelity and reenactment quality, especially in close-up regimes.

The comparison is also qualitative. GaussianAvatars is described as blurry and unable to model expression-dependent details well; GHA is described as improving dynamics but showing grid-like artifacts and blurriness; NPGA is described as synthesizing wrinkles but still appearing blurry on zoom-ins. ScaffoldAvatar is presented as especially strong in fine-detail reconstruction.

## 6. Ablations, scope, and methodological significance

The ablation study attributes the reported gains to three core ingredients: patch expressions, per-patch color modeling, and high-resolution optimization procedures. Removing patch expressions introduces artifacts around the lips and blurs details. Removing the per-patch color MLP reduces close-up sharpness. Replacing color-based densification with standard position-based densification hurts sharpness. Skipping progressive training slows convergence and reduces final quality [2507.10542].

These ablations matter because they isolate where the method departs from prior Gaussian avatar pipelines. Patch expressions are not presented merely as a higher-dimensional latent code; they are tied to a spatially localized geometric model and to localized Gaussian generation. Likewise, the color pathway is not globally shared but partitioned at patch level, and densification is guided by view-space color gradients rather than only geometric heuristics. A plausible implication is that ScaffoldAvatar’s improvements arise from aligning representational units, motion descriptors, and appearance predictors at the same local facial scale.

The scope of the method is equally clear from the reported pipeline. ScaffoldAvatar begins with multi-view capture, tracked meshes, and a patch-based geometric \(3\)D face model. This places it in the setting of high-fidelity performance-driven head avatars rather than text-to-avatar generation or marketplace asset retrieval. The term “scaffold” in ScaffoldAvatar refers to its use of Scaffold-GS anchors as a hierarchical scene representation coupled to patch-local facial dynamics, not to an abstract prompt scaffold.

In summary, ScaffoldAvatar defines a localized alternative to global-expression Gaussian avatars. Its stated contribution is the integration of \(432\) overlapping facial patches, \(8208\) patch expression parameters, patch-attached Scaffold-GS anchors, view-conditioned on-the-fly Gaussian synthesis, per-patch color prediction, color-based densification, and progressive \(1\)K-to-\(3\)K training into a real-time head avatar system aimed at close-up photorealism [2507.10542].

Source: https://www.emergentmind.com/topics/scaffoldavatar