---
title: 'FLAME-in-NeRF: Free-view Face Animation'
url: https://www.emergentmind.com/topics/flame-in-nerf
type: topic
---

# FLAME-in-NeRF: Free-view Face Animation

FLAME-in-NeRF refers to a family of neural rendering techniques that integrate the explicit, low-dimensional face modeling capacity of the FLAME (Faces Learned with an Articulated Model and Expressions) 3D morphable model (3DMM) with the high-fidelity, free-viewpoint scene synthesis capabilities of neural radiance fields (NeRF). This approach enables explicit control over a subject's facial expressions, identity, and pose within the neural radiance field, making it possible to synthesize photorealistic novel-view portrait videos with entangled scene and face manipulation. FLAME-in-NeRF leverages FLAME's compact parameterization as an input to the NeRF conditioning mechanism, and applies spatial priors to ensure disentanglement of facial and background regions, yielding a synthesis system capable of both high realism and direct parametric facial editing [2108.04913].

## 1. The Role of FLAME and 3D Morphable Models in Neural Radiance Fields

FLAME is a widely adopted 3DMM designed to generate fully controllable human face meshes. The model parameterizes facial shape, expression, and pose as follows:

- $\beta \in \mathbb{R}^{k_\beta}$: identity (shape) coefficients
- $\psi \in \mathbb{R}^{k_\psi}$: expression coefficients
- $\theta \in \mathbb{R}^{k_\theta}$: pose parameters (jaw, neck, eyeballs)

The output $M(\beta, \psi, \theta)$ is a triangular mesh encoding the 3D geometry of the face for a particular identity-expression-pose configuration.

In FLAME-in-NeRF, the FLAME parameterization enables a low-dimensional "expression code" $e \in \mathbb{R}^L$ for each video frame or sequence. This code serves as a control input, allowing the radiance field to deform and relight the head in accordance with arbitrary facial expressions. The explicit mesh provided by FLAME provides a structural prior that can be leveraged by the radiance field [2108.04913].

## 2. Neural Radiance Field Conditioning with Expression Codes

The standard NeRF formulation represents the scene as a continuous function parameterized by a multi-layer perceptron (MLP):

\[
\rho_\theta(x, d) \rightarrow (\sigma, c)
\]

where $x \in \mathbb{R}^3$ is the spatial location, $d \in \mathbb{S}^2$ the view direction, $\sigma$ the volume density, and $c$ the RGB color.

In FLAME-in-NeRF, the output of the volumetric MLP is conditioned not only on $x$ and $d$, but also on the FLAME-derived expression code $e$:

\[
\text{Density network:} \quad f_\sigma(x, e) \rightarrow \sigma(x,e)
\]
\[
\text{Color network:} \quad f_c(x, d, e) \rightarrow c(x, d, e)
\]

Expression control is injected using a feature-wise linear modulation (FiLM) mechanism. The code $e$ is mapped by a small MLP $M$ to $z = M(e)$, and within each block of the NeRF architecture, the activations are modulated:

\[
h' = \gamma(z) \odot h + \beta(z)
\]

where $(\gamma, \beta)$ are learnable projections, and $h$ is the intermediate feature vector.

This direct parametric conditioning, as opposed to the indirect deformation-field approach (e.g., Nerfies), enables the system to synthesize new facial configurations at inference by adjusting $e$ alone, enabling explicit, disentangled control over facial animation [2108.04913].

## 3. Spatial Priors and Disentanglement of Face and Background

A central challenge in conditioning NeRF with facial expression codes is to prevent entanglement of head and background appearance. Without constraints, the network may encode expression-dependent variation in the scene background, violating the desired separation.

FLAME-in-NeRF introduces a spatial prior through a binary or soft occupancy field $O(x; e_\text{ref})$, generated by ray marching the canonical FLAME mesh (in reference expression) into the world frame. The occupancy $O(x)$ masks the face region. To enforce head/background disentanglement, the network attenuates or nullifies the FiLM modulation for points outside the facial region by multiplying the modulation scale $\gamma(z)$ by $O(x)$:

- In the background ($O(x) \approx 0$), expression code dependence is zeroed, ensuring the background remains invariant to facial expression changes.
- In the face region ($O(x) \approx 1$), full modulation is applied, allowing the network to model expression-dependent appearance and geometry [2108.04913].

This spatial gating approach guarantees that varying the FLAME code only influences the synthesized face, not the underlying scene, yielding more robust and semantically meaningful controls for synthesis and animation.

## 4. Training Data, Loss Functions, and Optimization

The FLAME-in-NeRF pipeline requires minimal data acquisition: a 10–30 second handheld portrait video, with moderate head movement, recorded on a standard mobile device. Preprocessing involves:

- Camera pose estimation for every frame using structure-from-motion (e.g., COLMAP).
- Per-frame FLAME fitting to extract the expression code $e_t$ and a canonical head mesh.

The loss functions typically include:

- Photometric loss:
  \[
  L_\text{photo} = \sum_r \| C(r) - C_\text{gt}(r) \|^2_2
  \]
  ensuring synthesis matches the input images at the pixel level.
- Optional perceptual loss (e.g., VGG-based $L_\text{perc}$).
- Tikhonov regularization on expression codes: $L_\text{reg}(e) = \lambda_e \|e\|^2_2$.
- Disentanglement loss:
  \[
  L_\text{disent} = \lambda_\text{dis} \sum_{x \notin \text{face}} \left( \|\sigma(x, e_1) - \sigma(x, e_2)\|_1 + \|c(x, d, e_1) - c(x, d, e_2)\|_1 \right)
  \]
  enforcing invariance in the background region under changes in $e$.
- FLAME fitting loss: $L_\text{fit} = \| \hat{e}_t - e_t \|^2_2$.

Optimization typically proceeds with Adam for 150k–300k steps with an initial learning rate of $5 \times 10^{-4}$, decayed over training, yielding convergence in about 2–3 days on a Tesla V100. Per iteration, the system samples ∼1024 rays and 64 depths per ray [2108.04913].

## 5. Qualitative and Quantitative Outcomes

FLAME-in-NeRF supports explicit modification of facial expressions by reparameterizing the expression code $e$ at test time, which can be applied to synthesize:

- Expression transfer (e.g., transferring a smile onto a neutral head at a novel viewpoint).
- Arbitrary editing (e.g., combined brow raise and jaw drop in unseen angles).

Quantitative evaluation employs PSNR, SSIM, and LPIPS metrics on held-out test frames, as well as expression accuracy via correlations between rendered and ground-truth action units (AUs). Compared to baseline methods:

- Typical improvements include +2–3 dB PSNR over deformable NeRFs (e.g., Nerfies) and +0.05 SSIM, indicating higher photometric and structural fidelity.
- Qualitative improvements include sharper, more realistic facial animation and background consistency compared to deferred neural rendering methods that lack free view synthesis.

These outcomes reflect FLAME-in-NeRF's capacity to bridge detailed volumetric synthesis with interpretable, parametric face control [2108.04913].

## 6. Limitations and Failure Cases

Known constraints of FLAME-in-NeRF, as reported, include:

- Poor generalization to extreme expressions not represented in the training data, sometimes producing rendering artifacts such as "teeth popping" or volume collapse.
- Temporal artifacts (e.g., ghosting) in the background under rapid expression dynamics, due to the underlying NeRF static-world assumption.
- Imperfect rendering of highly specular regions (e.g., teeth, eyeglasses), manifesting as subtle flicker at novel viewpoints.
- Degraded head synthesis during long-range pose changes not covered by the training capture.

These limitations stem from a combination of the representational capacity of the NeRF, the expressivity of the FLAME prior, and the constraints imposed by limited, video-only capture [2108.04913].

## 7. Relationship to Related Approaches

FLAME-in-NeRF represents a distinct alternative to direct mesh-texture pipelines and implicit-deformation NeRF variants:

| Method                     | Expression Control | High-Frequency Detail | Background Synthesis |
|----------------------------|-------------------|----------------------|---------------------|
| Vanilla NeRF               | No                | Yes                  | Yes                 |
| Nerfies (deformable NeRF)  | Indirect (warp)   | Yes                  | Yes                 |
| Deferred Neural Rendering  | Yes (limited)     | Yes                  | No                  |
| FLAME-in-NeRF              | Explicit (3DMM)   | Yes                  | Yes                 |

Unlike methods relying on direct mesh deformation or global deformation fields, FLAME-in-NeRF achieves explicit, low-dimensional, and semantically meaningful face control without sacrificing background realism. The spatial prior enforced by the FLAME occupancy field distinguishes FLAME-in-NeRF from approaches that suffer from facial-background entanglement or lack free-view background synthesis [2108.04913].

## References

- "FLAME-in-NeRF: Neural control of Radiance Fields for Free View Face Animation" [2108.04913]

Source: https://www.emergentmind.com/topics/flame-in-nerf