Neural Parametric Head Models (NPHMs)

• Neural Parametric Head Models (NPHMs) are continuous implicit 3D representations that use high-dimensional latent codes to capture detailed facial identity and expression.
• They utilize signed distance functions to explicitly recover surface normals and extract mesh topology, enabling accurate reconstruction of subtle facial features.
• Advanced architectures like Pix2NPHM integrate vision transformers and mixed 2D/3D supervision to robustly estimate latent codes from single images, delivering state-of-the-art accuracy.

Neural Parametric Head Models (NPHMs) are continuous, implicit 3D head models parameterized by high-dimensional latent codes, enabling the synthesis of detailed head geometry surpassing mesh-based 3D morphable models. NPHMs, such as MonoNPHM, encode identity and expression in separate latent vectors and represent surface geometry through signed distance functions (SDF), allowing explicit recovery of surface normals and mesh topology. Accurate estimation of NPHM latent codes from visual data, particularly single 2D images, presents significant challenges due to ill-posedness, the high dimensionality of the latent space, and the entanglement of identity and nonrigid expression. Feed-forward methods based on vision transformers, notably Pix2NPHM, have recently enabled robust and scalable single-image NPHM regression, integrating geometric transformer backbones, mixed 2D/3D supervision, and optional inference-time optimization for high-fidelity face reconstruction (Giebenhain et al., 19 Dec 2025).

1. Model Formulation and Latent Space

An NPHM defines a continuous mapping from 3D query points to SDF values:

$$\mathscr{F}_{\mathrm{NPHM}}(x; z_{\mathrm{id}}, z_{\mathrm{ex}}) \mapsto \phi(x)$$

where $x \in \mathbb{R}^3$ is a spatial location, and $z_{\mathrm{id}}$, $z_{\mathrm{ex}}$ are high-dimensional codes controlling rigid identity and nonrigid expression, respectively. The surface $\mathcal{S}$ is implicitly defined as the zero level set $\{x \mid \phi(x) = 0\}$; surface normals are $\nabla_x \phi$, and mesh extraction utilizes marching cubes.

The expressiveness of the latent codes enables subtle geometric detail, including person-specific features (e.g., cheek dimples, expression-induced wrinkles). However, the inverse problem—inferring $(z_{\mathrm{id}}, z_{\mathrm{ex}})$ from 2D images—is under-constrained: many code values yield similar projections under variable lighting or occlusion, and small code perturbations can produce high-frequency geometric variation.

2. Network Architecture: Pix2NPHM

Pix2NPHM addresses NPHM regression using a feed-forward transformer pipeline with a geometric vision transformer backbone. Two separate ViT encoders pretrained on geometry-specific tasks provide token representations:

• $\mathscr{E}_n$: predicts per-pixel surface normals
• $\mathscr{E}_p$: predicts canonical point positions

Given input image $I$, each encoder yields a sequence of tokens:

$$\mathscr{E}_{geo}(I) \in \mathbb{R}^{L \times D}, \quad geo \in \{n, p\}.$$

Tokens from both encoders are concatenated, and trainable classifier tokens

$$T_{\mathrm{CLS}} = \left\{ T^{\mathrm{ex}}, T^{\mathrm{id}_1}, \ldots, T^{\mathrm{id}_{65}} \right\}$$

(one global expression, 65 local identity) are appended. Over $K$ transformer layers, the structure enables multi-head self-attention among geometric cues and classifier tokens.

Outputs are read out via MLPs:

$$z_{\mathrm{id}} = \mathrm{MLP}_{\mathrm{id}}(T'^{\mathrm{id}_1}, \ldots, T'^{\mathrm{id}_{65}}), \quad z_{\mathrm{ex}} = \mathrm{MLP}_{\mathrm{ex}}(T'^{\mathrm{ex}})$$

which parameterize the MonoNPHM decoder. Architectural pretraining on geometric tasks substantially improves 3D generalization compared to generic encoders.

3. Supervision, Losses, and Training Regimen

Pix2NPHM is trained with a datasheet-mixed paradigm integrating direct 3D SDF supervision and 2D normal-based self-supervision:

• 3D SDF loss for registered head scans:

$$\mathcal{L}_{3D} = \frac{1}{N} \sum_{i=1}^N \left| \phi(x_i; \hat{z}_{\mathrm{id}}, \hat{z}_{\mathrm{ex}}) - \phi(x_i; z_{\mathrm{id}}^{gt}, z_{\mathrm{ex}}^{gt}) \right|_1$$

Ground-truth codes $(z_{\mathrm{id}}^{gt}, z_{\mathrm{ex}}^{gt})$ are established by energy-minimization registration of scan surfaces to the canonical MonoNPHM template. Supervision in SDF space is critical, as direct latent code or $L_2$ loss is non-convergent due to code ambiguity.

• 2D normal-based loss for in-the-wild frames with no 3D annotation:

$$\mathcal{L}_{2D}^n = \sum_{p \in \mathcal{P}} \Big(1 - \langle \mathrm{Render}_\pi(\phi; \hat{z}_{\mathrm{id}}, \hat{z}_{\mathrm{ex}})_p, I^n_p \rangle \Big)$$

Here, $I^n$ are pseudo ground-truth normals produced by a pretrained decoder, and rendering is NeuS-style volumetric projection under estimated cameras. This loss is robust to lighting and occlusion, unlike pixel-based photometric losses.

• Latent regularization encourages code scale and smoothness:

$$\mathcal{R}(z_{\mathrm{id}}, z_{\mathrm{ex}}) = \lambda_{\mathrm{id}} \|z_{\mathrm{id}}\|_2^2 + \lambda_{\mathrm{ex}} \|z_{\mathrm{ex}}\|_2^2$$

The total loss on each mini-batch is:

$$\mathcal{L}_{total} = \lambda_{3D} \mathcal{L}_{3D} + \lambda_{2D} \mathcal{L}_{2D}^n + \lambda_{reg} \mathcal{R}(z_{\mathrm{id}}, z_{\mathrm{ex}})$$

4. Data Curation and Annotation Pipeline

Supervision leverages both 3D scan data and large 2D video collections. For the 3D regime, approximately 100,000 high-resolution head scans from public datasets (FLAME, LYHM, BU-3DFE) are unified to a shared reference frame via FLAME registration. MonoNPHM codes for each scan are optimized by minimizing

$$\sum_{x \in \mathcal{X}} |\phi(x; z_{\mathrm{id}}, z_{\mathrm{ex}}) - \phi_{gt}(x)|_1 + \mathcal{R}(z_{\mathrm{id}}, z_{\mathrm{ex}})$$

Estimates serve as explicit targets for $\mathcal{L}_{3D}$.

For 2D weak supervision, tens of thousands of frames (CelebV-HQ, VoxCeleb, AffectNet) are processed using a FLAME tracker to estimate camera poses; normals are generated via the pretrained normal decoder. This dataset diversity ensures broad coverage of shape, appearance, and unconstrained conditions.

5. Inference and Optional Test-Time Optimization

During inference, Pix2NPHM produces a first-pass estimate $(\hat{z}_{\mathrm{id}}, \hat{z}_{\mathrm{ex}})$ and camera $\hat{\pi}$ via feed-forward prediction. For additional fidelity, the predicted codes and camera may be refined by optimization:

$$\arg\min_{z_{\mathrm{id}}, z_{\mathrm{ex}}, \pi} \lambda_n \mathcal{L}_{2D}^n + \lambda_p \mathcal{L}_{2D}^p + \lambda_{reg} \left\| \begin{bmatrix} z_{\mathrm{id}} \ z_{\mathrm{ex}} \end{bmatrix} - \begin{bmatrix} \hat{z}_{\mathrm{id}} \ \hat{z}_{\mathrm{ex}} \end{bmatrix} \right\|_2^2$$

where $\mathcal{L}_{2D}^p$ is an $L_1$ penalty on rendered canonical point maps. This Levenberg–Marquardt (or Adam) optimization is initialized at the feed-forward prediction and converges quickly, providing a geometric refinement that sharpens local detail without divergence.

6. Empirical Evaluation and Comparison

The Pix2NPHM framework achieves state-of-the-art accuracy on single-image 3D face reconstruction:

Benchmark	Method	L1 (mm)	L2 (mm)	Normal Consistency	Runtime
NeRSemble SVFR (posed)	Pix2NPHM (ffwd)	1.55	1.05	0.894	~8 fps (RTX 3080)
	Pix2NPHM (+opt.)	1.37	0.92	0.897	~85 s / image
	SHeaP (prev best FF)	~2.08	~1.41	~0.876	N/A
NoW (neutral)	Pix2NPHM (ffwd)	0.83*	1.03	-	~8 fps
	Pix2NPHM (+opt.)	0.81*	1.01	-	~85 s / image
	Best prior public	-	~1.07	-	-

(* median errors reported)

Relative to prior approaches—in particular, FLAME-based (DECA, MICA, EMOCA₂, TokenFace, SHeaP) and even optimized FLAME (FlowFace, MetricalTracker, Pixel3DMM)—Pix2NPHM surpasses both direct and optimization-based methods in L1/L2 and normal-consistency error. Notably, it reduces error by more than 30% over MonoNPHM's pure photometric fitting.

Qualitatively, Pix2NPHM demonstrates resilience to real-world noise: it can reconstruct details obscured by challenging lighting, occlusions (hands, hair), and accessories. Its geometric refinement sharpens high-frequency content at semantic locations (mouth, eyes, nasolabial folds), as observed in overlay visualizations (Giebenhain et al., 19 Dec 2025).

7. Significance and Impact

NPHMs, realized through Pix2NPHM, establish a new paradigm for interpretable, high-fidelity human head reconstruction from monocular imagery. The integration of geometry-focused ViT pretraining, 3D implicit field supervision, and combined weakly/strongly labeled data enables scalable, robust fitting across unconstrained data. This suggests that future parametric models for human faces will increasingly merge implicit neural fields with advanced transformer architectures, shifting away from mesh-based 3DMMs for tasks requiring high geometric expressivity.

Pix2NPHM constitutes the first demonstration of interactive, automatic single-image NPHM regression, closing the gap with supervised mesh models while retaining the modeling flexibility of implicit SDF representations. Its pipeline sets a reference for subsequent work aiming to bridge vision transformer architectures and neural implicit 3D morphable modeling (Giebenhain et al., 19 Dec 2025).