GAT-NeRF: Geometry-Aware Transformer NeRF

• The paper introduces a novel framework that integrates explicit geometric priors with Transformer-based feature aggregation to enhance NeRF’s ability to reconstruct dynamic 4D facial avatars.
• It employs a coordinate-aligned MLP along with multi-modal encoding of 3D spatial data, 3DMM expressions, and learnable latent codes to achieve state-of-the-art performance.
• The approach significantly improves the synthesis of fine facial details such as dynamic wrinkles and textures, as evidenced by reduced L1 error and improved SSIM metrics.

Geometry-Aware-Transformer Enhanced NeRF (GAT-NeRF) is a neural scene representation framework that advances high-fidelity 4D facial avatar reconstruction from monocular video by integrating explicit geometric priors into the Neural Radiance Fields (NeRF) paradigm using Transformer-based architectures. GAT-NeRF addresses the challenge of capturing high-frequency facial details, such as dynamic wrinkles and subtle textures, by synergistically combining a coordinate-aligned multilayer perceptron (MLP) with a Geometry-Aware Transformer (GAT) module. This architecture effectively fuses multi-modal features—including 3D spatial coordinates, 3D Morphable Model (3DMM) expression parameters, and learnable latent codes—enabling enhanced geometric and photorealistic avatar synthesis driven by highly information-constrained input streams (Chang et al., 21 Jan 2026).

1. Foundations: Neural Radiance Fields and Geometric Priors

NeRF represents scenes as continuous volumetric functions, $F_\Theta((x, d)) = (\sigma, c)$, where $x \in \mathbb{R}^3$ denotes spatial position, $d \in \mathbb{S}^2$ specifies viewing direction, $\sigma$ is volume density, and $c$ is RGB color. Rendering follows the classic volume rendering equation:

$$C(r) = \int_{t_n}^{t_f} T(t)\, \sigma(x(t), d)\, c(x(t), d)\, dt$$

with transmittance $$T(t) = \exp\left(-\int_{t_n}^t \sigma(x(s), d) ds\right)$$. Efficient approximation is achieved via stratified and hierarchical sampling schemes, utilizing coarse and fine MLP networks. Despite its effectiveness, standard NeRF pipelines have limited ability to encode fine parametric facial motion or high-frequency texture solely from monocular imagery, motivating architectural enhancements through geometric priors and advanced feature aggregation (Chang et al., 21 Jan 2026).

2. Architecture: Multi-Modal Encoding and Geometry-Aware Transformer

GAT-NeRF receives, for each sampled 3D point $p$ and frame index $i$:

• Spatial coordinates $p \in \mathbb{R}^3$, encoded using a positional encoding $PE(p)$ with $L_p = 10$ frequency bands, yielding 63 dimensions.
• Expression parameters $\delta \in \mathbb{R}^{76}$ sourced from a 3D Morphable Model (3DMM).
• Learnable latent codes $\gamma_i \in \mathbb{R}^{32}$, unique per frame.

These features are concatenated into $$X_{\mathrm{concat}} = [PE(p) \ \|\ \delta \ \|\ \gamma_i] \in \mathbb{R}^{171}$$. A linear projection maps $X_{\mathrm{concat}}$ to the Transformer's hidden space ($D=256$):

$X_{\mathrm{proj}} = X_{\mathrm{concat}} \cdot W_p$

Standard Query, Key, and Value matrices ($N_h=8$ heads, $d_k=32$) enable multi-head self-attention within a single Transformer encoder layer. The inclusion of explicit geometry via 3DMM and spatial encoding allows the Transformer's self-attention mechanism to leverage geometric locality and contextual cues across temporal frames and expression states (Chang et al., 21 Jan 2026).

3. Coordinate-Aligned MLP Stack and Volume Rendering

Subsequent to Transformer-based feature aggregation, the output ($z \in \mathbb{R}^{256}$) feeds into a five-layer, 256-unit MLP for density prediction. At the third layer, a skip connection reintegrates $X_{\mathrm{concat}}$, enhancing representational fidelity. The resulting feature vector ("feat," Editor's term) is passed to a density head ($\sigma = \mathrm{Linear}(\mathrm{feat})$) and, together with the viewing direction encoding $PE(d)$ (4 bands, 27 dimensions), to a color MLP that infers view-dependent color $c$.

Rendering aggregates these pointwise predictions via volumetric integration, matching NeRF's background, but GAT-NeRF's per-point features are geometrically enriched by Transformer and MLP interplay (Chang et al., 21 Jan 2026).

4. Optimization Objectives and Regularization

Training is supervised with a photometric reconstruction loss, comparing synthesized ray colors $\hat{C}(r; \Theta, \gamma_i)$ against ground-truth $C_{\mathrm{gt}}(r)$:

$$L_{\mathrm{photo}, i}(\Theta, \gamma_i) = \sum_{r \in R_i} \| \hat{C}(r; \Theta, \gamma_i) - C_{\mathrm{gt}}(r) \|_2^2$$

where $R_i$ is the set of rays for frame $i$. Latent codes are $\ell_2$-regularized ($\lambda_{\gamma} = 0.05$). The total loss, applied to both coarse and fine NeRF networks, sums:

$$L_{\mathrm{total}} = \sum_{i=1}^K [ L_{\mathrm{photo}, i}(\theta_c, \gamma_i) + L_{\mathrm{photo}, i}(\theta_f, \gamma_i) + \lambda_{\gamma} \|\gamma_i\|_2^2 ]$$

No explicit geometric or perceptual losses are introduced; the GAT module's structure is relied upon to foster detail synthesis (Chang et al., 21 Jan 2026).

5. Implementation Details and Hyperparameters

Key hyperparameters and workflow specifications include:

• Dataset: NeRFace monocular video (cropped to $512 \times 512$; 90% train, 10% test).
• Batch: 1024 rays per iteration.
• Volumetric sampling: $N_c = 64$ (coarse), $N_f = 64$ (fine) per ray.
• Optimizer: Adam, learning rate $3 \times 10^{-4}$, 300k iterations.
• Transformer: 1 encoder layer, $d_{\mathrm{model}} = 256$, 8 heads, $d_{\mathrm{ffn}} = 2048$.
• Latent codes: $\gamma_i \in \mathbb{R}^{32}$.
• Positional encodings: $p$ (10 bands), $d$ (4 bands) (Chang et al., 21 Jan 2026).

6. Quantitative and Qualitative Evaluation

On NeRFace (self-reenactment), GAT-NeRF demonstrates state-of-the-art performance, as summarized:

Method	L1 (↓)	PSNR (↑)	SSIM (↑)	LPIPS (↓)
NeRFace [Gafni 21]	0.047	24.092	0.926	0.074
PointAvatar [Zheng 23]	0.015	27.000	0.915	0.069
FlashAvatar [Xiang 24]	0.015	26.883	0.916	0.071
GAT-NeRF	0.013	24.822	0.932	0.070

Ablation studies validate the impact of the GAT module, which reduces L1 error (0.047→0.012) and improves SSIM (0.926→0.936). Perceptual LPIPS is further improved by adding latent codes (0.080→0.070), albeit with minor PSNR tradeoff. Qualitative assessments show enhanced synthesis of dynamic wrinkles and skin texture (e.g., periocular wrinkles, forehead furrows, acne scars), enabling realistic pose/expression decoupling and cross-identity expression transfer (Chang et al., 21 Jan 2026).

7. Relation to Broader Geometric Transformer Pipelines

GAT-NeRF shares thematic similarities with works such as GeoNeRF (Johari et al., 2021), which integrate geometry priors and Transformer-based attention to enhance view synthesis performance. While GeoNeRF employs a two-stage geometry reasoner and a multi-scale Transformer to aggregate stereo cost volumes across multi-view inputs, GAT-NeRF is specifically tuned for monocular, dynamic 4D facial avatar construction, incorporating 3DMM priors and latent scene codes for per-frame refinement. Both approaches support the trend of leveraging Transformer feature learning for high-fidelity scene reconstruction, but GAT-NeRF distinguishes itself via its focus on temporal expressivity, explicit morphable model input, and control for avatar applications (Johari et al., 2021).

---

In summary, GAT-NeRF advances the neural rendering field by harnessing geometry-aware attention, explicit multi-modal input fusion, and a lightweight Transformer architecture embedded within the canonical NeRF framework. This delivers high-precision, photorealistic, temporally controllable 4D facial avatars compatible with stringent multimedia and virtual human requirements (Chang et al., 21 Jan 2026).