---
title: 'MVCHead: 3D Gaussian Head Avatar Generation'
url: https://www.emergentmind.com/topics/mvchead-architecture
type: topic
---

# MVCHead: 3D Gaussian Head Avatar Generation

MVCHead is a multi-view consistent 3D Gaussian head avatar generation architecture designed to produce high-fidelity 3D head models using only randomly sampled 2D images, explicitly eliminating the need for multi-view data, 3D supervision, or intermediate view synthesis. Leveraging a single-shot state space model, MVCHead directly enforces multi-view consistency within the 3D representation by regressing a set of anisotropic 3D Gaussians under strict structural constraints. Its central innovations include the hierarchical refinement of Gaussians through a Hierarchical State Space (HiSS) block—featuring the novel Hierarchical Bi-directional State Scan (HiBiSS) mechanism—and a dedicated SE(3) Multi-view Critic that scores view consistency in the absence of real multi-view pairs [2605.25220].

## 1. Pipeline and Representation

The core pipeline of MVCHead inputs a 512-dimensional Gaussian latent code $z \sim N(0, I)$ and, during training, a set of $K$ camera poses $\{T_k\}_{k=1}^{K}$ sampled from a canonical front-hemisphere rig. The model outputs a set of $N = 240{,}000$ anisotropic 3D Gaussians as $S_\theta(z) = \{g_i = (\mu_i, s_i, q_i, \alpha_i, c_i)\}_{i=1}^N$, where $\mu_i \in \mathbb{R}^3$ is the center, $s_i \in \mathbb{R}_+^3$ the scales, $q_i \in \mathbb{H}$ a rotation quaternion, $\alpha_i \in (0,1)$ opacity, and $c_i \in [0,1]^3$ color. A differentiable renderer $\mathcal{R}$ maps this set and a given pose $T$ to an RGB image $I \in \mathbb{R}^{H\times W\times 3}$.

Processing follows a hierarchical sequence:
1. **Mapping Network**: $z$ is mapped to $w \in W$ via an MLP, following a StyleGAN-like paradigm.
2. **Initial Token Scaffold**: Learnable tokens $T^0 \in \mathbb{R}^{P\times d_0}$ (with $P \approx 512$, $d_0 = 3$) are lifted into a dense $H\times W\times d$ feature grid $F^0$ using multi-frequency positional encoding.
3. **Appearance Conditioning**: At each resolution level $l$, $F^l$ is modulated by AdaIN parameters $(\gamma(w), \beta(w))$ derived from $w$, facilitating disentanglement of geometry and appearance.
4. **Hierarchical State Space (HiSS) Blocks**: A cascade of $L$ levels, each upsamples the Gaussian population and refines parameters via dual-mixing (self-attention and HiBiSS scans) and anchor-based per-attribute MLP heads.
5. **3DGS Renderer**: The concatenated set of all-level Gaussians $S_\theta(z)$ is rendered into $K$ image views under the camera poses $\{T_k\}$.
6. **Critics**: Training utilizes an adversarial texture discriminator $D_\phi$ (camera-conditioned GAN loss) and an SE(3)-equivariant Multi-view Critic $E_\psi$ to ensure cross-view consistency.

## 2. Hierarchical State Space (HiSS) and Parameter Regression

Each HiSS block processes the $H\times W\times d$ feature grid, doubling or quadrupling (upsample ratio $r=2$–$4$) the number of Gaussians at each level. The dual-mixer structure operates as follows:

- **Branch 1**: Multi-Head Self-Attention (with MLP and LayerNorm), designed to aggregate global, non-axis-aligned dependencies for both shape and identity cues.
- **Branch 2**: The HiBiSS state-space mixer, enforcing local, axis-aligned consistency via dedicated recurrent mechanisms.

After mixing, the fused features $F^l_{\mathrm{out}} \in \mathbb{R}^{H \times W \times d}$ are dispatched to per-attribute MLP heads $($MLP$_\mu$, MLP$_s$, MLP$_q$, MLP$_\alpha$, MLP$_c)$, each regressing spatial offsets:

\[
\begin{aligned}
\Delta \mu_k^l &= \mathrm{MLP}_\mu(F^l_{\mathrm{out}}[i,j]) \\
\mu_k^l &= \mu_{p(k)}^{l-1} + \Delta\mu_k^l \\
\Delta s_k^l &= \mathrm{MLP}_s(F^l_{\mathrm{out}}[i,j]) \\
s_k^l &= \text{softplus}(s_{p(k)}^{l-1} + \Delta s_k^l) \\
\Delta q_k^l &= \mathrm{MLP}_q(F^l_{\mathrm{out}}[i,j]) \\
q_k^l &= \text{normalize}(q_{p(k)}^{l-1} \oplus \Delta q_k^l) \\
\alpha_k^l &= \sigma(\mathrm{MLP}_\alpha(F^l_{\mathrm{out}}[i,j])) \\
c_k^l &= \sigma(\mathrm{MLP}_c(F^l_{\mathrm{out}}[i,j]))
\end{aligned}
\]

where $p(k)$ denotes the parent anchor index and $\oplus$ is quaternion composition in tangent space, re-normalized. The generic 2D state-space recurrence within HiSS is:

\[
h_{t+1} = A\,h_t + B\,x_t,\quad y_t = C\,h_t + D\,x_t
\]

with $A,B,C,D\in\mathbb{R}^{d\times d}$.

## 3. Hierarchical Bi-directional State Scan (HiBiSS)

HiBiSS extends the Mamba 1D scan into four axis-aligned 2D recurrences—$(\to, \leftarrow, \downarrow, \uparrow)$—targeting the axes most affected by yaw/pitch-induced view drift. For grid $F \in \mathbb{R}^{H \times W \times d}$, the horizontal forward scan operates as:

\[
h^{\to}_{i, j+1} = A_h\,h^{\to}_{i, j} + B_h\,F_{i, j},\quad \tilde{F}^{\to}_{i, j} = C_h\,h^{\to}_{i, j} + D_h\,F_{i, j}
\]

Analogous recurrences apply for the other three directions, each with learned weights. After all four scans, outputs are fused as:

\[
\tilde{F}_{i,j} = W_{\to} \tilde{F}^{\to}_{i,j} + W_{\leftarrow} \tilde{F}^{\leftarrow}_{i,j} + W_{\downarrow} \tilde{F}^{\downarrow}_{i,j} + W_{\uparrow} \tilde{F}^{\uparrow}_{i,j}
\]

where $W_\cdot$ are learned fusion matrices. LayerNorm and FFN are applied before the outcome is merged with the attention branch.

This multi-directional noncausal scheme efficiently aligns local 3D features, minimizing occurrence of multi-view inconsistencies along view-sensitive axes.

## 4. SE(3) Multi-view Critic and Losses

The SE(3) Multi-view Critic $E_\psi$ evaluates K-tuple image sets and their corresponding poses for 3D consistency:

\[
s = E_\psi(\{\hat{I}_k\}_{k=1}^K, \{T_k\}_{k=1}^K) \in \mathbb{R}
\]

Its architecture comprises a ViT-style encoder (patch tokenization + self-attention) and a Geometric Transform Attention module. In GTA, queries and keys are pre-rotated by relative extrinsic matrices $\tilde{T}_k = T_k T_1^{-1}$ within $\mathrm{SO}(3)$ to ensure equivariance under rigid transformations and invariance to camera intrinsics. Output tokens are pooled by a small MLP into a scalar logit.

E$_\psi$ is pre-trained to distinguish positive sets (all K renders from the same latent $z$) from negative sets (distinct latents, same poses) via the cross-entropy loss:

\[
\mathscr{L}_{\mathrm{critic}} = -\mathbb{E}_{S^+}[\log \sigma(s)] - \mathbb{E}_{S^-}[\log(1-\sigma(s))]
\]

After convergence, the critic is frozen and used to formulate a differentiable reward for the generator:

\[
\mathscr{L}_{mvc} = -\mathbb{E}_{z, \{T_k\}} \left[E_\psi(\{\mathcal{R}(S_\theta(z), T_k)\}, \{T_k\})\right]
\]

GAN loss for per-view adversarial supervision is also imposed:

\[
\mathscr{L}_{adv} = \mathbb{E}_z \frac{1}{K}\sum_{k=1}^K \mathrm{softplus}\left(-D_\phi(\mathcal{R}(S_\theta(z),T_k), T_k)\right)
\]

Regularization terms $L_{knn}$ (penalizing sparse regions) and $L_{ctr}$ (penalizing centroid drift from parent anchors) further stabilize learning.

The total objective combines these components:

\[
\mathscr{L}_{total} = \lambda_{mvc} \mathscr{L}_{mvc} + \mathscr{L}_{adv} + \lambda_{knn} L_{knn} + \lambda_{ctr} L_{ctr}
\]

with typical values $\lambda_{mvc}=1.0, \lambda_{knn}=0.1, \lambda_{ctr}=0.1$.

## 5. Training, Hyperparameters, and Inference

MVCHead utilizes the following hyperparameter and training schedule:

- **Number of Gaussians**: $N_0=15\,\text{K}$ in the initial block, upsampled by $r=2$ per block over $L=4$ levels, resulting in $15\,\text{K} + 30\,\text{K} + 60\,\text{K} + 120\,\text{K} = 225\,\text{K} \approx 240\,\text{K}$.
- **Token grid**: $H \times W = 16 \times 16$ at all levels, $d=256$.
- **Learning rates**: Generator $\theta$: $1\mathrm{e}{-4}$, discriminator $\phi$: $2\mathrm{e}{-4}$, critic $\psi$: $1\mathrm{e}{-4}$, all with Adam ($\beta_1=0, \beta_2=0.99$).
- **Training steps**: $10^7$ iterations on $4 \times$ NVIDIA H100 GPUs ($\sim$3 days).
- **Rendering resolution**: $512 \times 512$.
- **Critic views**: $K=4$ per set.

The high-level training and inference procedure is codified by the following pseudocode:

```python
# Pretrain Critic Eψ
for iter in [1..I1]:
    sample z_batch, poses T_batch
    I⁺ = {ℛ(Sθ(z),T_k)}_k         # Positives
    I⁻ = {ℛ(Sθ(z_i),T_k)}         # Negatives (shuffled)
    L_critic = BCE(Eψ(I⁺,T),1) + BCE(Eψ(I⁻,T),0)
    update ψ to minimize L_critic

# Joint Training
for iter in [1..I2]:
    sample z_batch, poses {T_k}
    S = Sθ(z_batch)               # Hierarchical HiSS + HiBiSS
    I = {ℛ(S, T_k)}               # K renders
    L_adv = (1/K) ∑ softplus( -Dφ(I_k,T_k) )
    L_mvc = - Eψ(I, {T_k})
    L_total = λ_mvc*L_mvc + L_adv + λ_knn*L_knn + λ_ctr*L_ctr
    update θ to minimize L_total
    update φ with standard GAN steps

# Inference
z ∼ N(0,I)
S = Sθ(z)
render any pose T: I = ℛ(S,T)
```

## 6. FaceGS-10K Dataset and Downstream Applications

The FaceGS-10K dataset, released with MVCHead, comprises 10,000 high-fidelity 3D Gaussian head assets, each with 240,000 Gaussians and 24 precomputed $512 \times 512$ renders covering the frontal hemisphere. These assets are intended for scalable supervision of future 3D head modeling research and for privacy-preserving digital avatar generation.

A plausible implication is that such a dataset, being produced fully automatically and without 3D or multi-view supervision, could catalyze further research in consistent 3D generation from unconstrained 2D sources. The dataset directly reflects the performance ceiling of the MVCHead architecture under unpaired 2D supervision alone [2605.25220].

---

For detailed implementation, refer to the project repository and supplementary code provided at https://humansensinglab.github.io/MVCHead/. The complete methodological formulation, equations, and pipeline described above reflect the architecture as documented in [2605.25220].

Source: https://www.emergentmind.com/topics/mvchead-architecture