Head Group: 3D Avatars, Talking Heads & Transformers

• Head Group Page is a multifaceted research area that explores both rendered human heads and computational attention heads using explicit structural designs.
• High-quality 3D head avatars employ canonical implicit representations and deformation fields to achieve photorealism and temporal coherence.
• Disentangled talking-head synthesis and MEA in Transformers utilize geometric bottlenecks and head-level compositions to enhance controllability and performance.

In current arXiv literature, “head” names both the human head as a rendering and animation target and the attention head as a computational unit in Transformer models. Across these settings, recent work has concentrated on explicit structure: canonical spaces and deformation fields for photorealistic 3D heads, geometric bottlenecks and modulated decoders for talking-head synthesis, and head-level linear composition for inter-head interaction in LLMs. Representative examples are HQ3DAvatar for controllable 3D head avatars (Teotia et al., 2023), DisCoHead for audio-and-video-driven talking heads (Hwang et al., 2023), and Multi-head Explicit Attention for explicit interaction among attention heads (Peng et al., 27 Jan 2026).

1. High-quality controllable 3D head avatars

HQ3DAvatar formulates the target as a high-quality, controllable 3D head avatar that is trained from multi-view, high-resolution RGB video of one person, can be rendered from arbitrary viewpoints and unseen expressions, can be driven at test time by a monocular RGB video, and runs at interactive rates, up to real time at lower resolution (Teotia et al., 2023). The method is positioned against two earlier families. Mesh-based templates such as 3DMMs and FLAME provide explicit geometry and parametric control via expression and pose coefficients, but are described as limited by linear blendshapes and by difficulty with high-frequency skin detail, mouth interior, tongue, teeth, hair, and topological changes such as lips opening or closing. Voxel or cube-based volumetric primitives can provide strong photorealism and efficient pruning of empty space, but are described as hard to configure from photometric loss alone, prone to artifacts in highly detailed or sparsely observed regions, and expensive to train.

The central design choice is a canonical implicit NeRF-like representation with no geometric template. A learned canonical space is shared across frames, while a deformation field explains dynamics. The overall field is

$$A:(\mathbf{x},\mathbf{d},e)\rightarrow(\mathbf{c},\sigma),$$

with world-space point $\mathbf{x}\in\mathbb{R}^3$, viewing direction $\mathbf{d}\in\mathbb{S}^2$, expression and head-motion latent code $e\in\mathbb{R}^{256}$, RGB color $\mathbf{c}\in\mathbb{R}^3$, and volume density $\sigma\in\mathbb{R}_{\ge 0}$. The representation is factorized into a deformation network $A_\theta$, a multiresolution hash-grid encoder $A_\alpha$, and a radiance MLP $A_\beta$.

Canonicalization is realized by a residual warp,

$$\mathbf{x}_o = A_\theta(\gamma(\mathbf{x}), e) + \mathbf{x},$$

so the radiance field is defined in canonical space rather than directly in world coordinates. The paper reports that directly feeding $\mathbf{x}\in\mathbb{R}^3$0 into the radiance MLP without canonical deformation causes overfitting to training views and artifacts, particularly in moving regions such as the mouth, eyelids, and hair. This architecture therefore uses the deformation field to absorb geometric motion and leaves the canonical radiance field to represent appearance and fine detail.

2. Canonical conditioning, temporal consistency, and empirical behavior

The training setup uses multi-view video of one subject captured with 24 synchronized Sony RXO II cameras in a 360° rig at 4K and 25 fps; the main experiments use 18 views and 1500 frames per view at 960×540, with camera intrinsics and extrinsics from Metashape and background matting via Background Matting V2 (Teotia et al., 2023). For each image, a CNN image encoder based on pre-trained VGG-Face with an added linear layer produces a 256-dimensional latent code $\mathbf{x}\in\mathbb{R}^3$1, and all VGG layers plus the new layer are fine-tuned. This code conditions both the deformation network and the radiance network, allowing the model to represent expression-dependent appearance such as wrinkles during smiling and changes in mouth-interior illumination.

A distinctive component is the optical-flow-based canonical consistency loss,

$\mathbf{x}\in\mathbb{R}^3$2

constructed by computing dense 2D correspondences between consecutive frames, lifting them to 3D by expected depth, mapping both 3D points into canonical space, and penalizing their difference. The stated effect is stronger temporal coherence and fewer artifacts in under-constrained regions such as hair, eyelids, teeth, and tongue. The full objective is

$\mathbf{x}\in\mathbb{R}^3$3

with $\mathbf{x}\in\mathbb{R}^3$4 and $\mathbf{x}\in\mathbb{R}^3$5. Training uses 500k iterations, learning rates of $\mathbf{x}\in\mathbb{R}^3$6 for the encoder, $\mathbf{x}\in\mathbb{R}^3$7 for the deformation MLP, $\mathbf{x}\in\mathbb{R}^3$8 for the hash grid, and $\mathbf{x}\in\mathbb{R}^3$9 for the radiance MLP, plus error-map-based ray sampling and a 64³ binary occupancy grid.

Spatial detail is carried by an Instant-NGP-style multiresolution hash grid with 16 levels, up to $\mathbf{d}\in\mathbb{S}^2$0 entries per level, 2-dimensional features per entry, coarsest resolution $\mathbf{d}\in\mathbb{S}^2$1, and finest resolution $\mathbf{d}\in\mathbb{S}^2$2. The paper attributes to this design faster convergence and high-resolution rendering, including FHD and 2K examples. Reported performance is about 12 hours of training on a single A100, roughly 10 fps at 960×540, slightly below 3 fps at 1920×1080, and about 25 fps at 480×270. On held-out views consisting of 2 cameras and 200 frames each, HQ3DAvatar reports PSNR 31.23, L1 2.79, SSIM 0.884, and LPIPS 0.113, compared with HyperNeRF++ at 26.42/5.61/0.851/0.172, MVP at 28.72/3.64/0.828/0.143, and NeRFBlendshape++ at 29.66/3.23/0.875/0.133. The method is explicitly person-specific, assumes controlled lighting, and still exhibits artifacts in occlusion and disocclusion events such as a tongue transitioning from inside to outside the mouth.

3. Disentangled talking-head generation

DisCoHead addresses audio-and-video-driven talking-head generation conditioned on a source identity frame $\mathbf{d}\in\mathbb{S}^2$3, a head-driving video frame $\mathbf{d}\in\mathbb{S}^2$4, speech audio aligned to the target frame, and a separate eye-region driving video (Hwang et al., 2023). The stated objective is to preserve the source identity, follow the head pose of the head-driving video, lip-sync to the audio, and reproduce eye blinks and eyebrow motion from the eye-region video. The method explicitly targets disentangled control of head pose or global head motion, articulatory facial expressions driven by audio, and non-speech facial expressions in the eye region.

The architecture has three components: a Head-Motion Predictor, a Motion-Aware Encoder, and an Expression-Control Decoder. Head motion is isolated through a single geometric transformation used as a bottleneck. This transformation can be either affine or thin-plate spline (TPS), and the paper states that both work well as geometric bottlenecks to isolate and steer head motion. In the affine case, a PCA-based predictor adapted from MRAA estimates a relative transformation

$\mathbf{d}\in\mathbb{S}^2$5

In the TPS case, a smooth nonrigid warp is defined by

$\mathbf{d}\in\mathbb{S}^2$6

The key architectural constraint is that this single transformation is the only pathway through which head-pose information from $\mathbf{d}\in\mathbb{S}^2$7 enters the generator.

Expression control is separated from geometry. Audio is encoded from an audio spectrogram spanning a 400 ms window centered on each frame using four 1D convolution layers, an LSTM, and two fully connected layers. A masked eye-region frame is encoded by a dedicated eye encoder with five downsample blocks, global average pooling, and two fully connected layers. The resulting audio and eye features are concatenated into $\mathbf{d}\in\mathbb{S}^2$8 and injected through StyleGAN2-style weight modulation in the decoder. This places mouth or jaw motion under audio control and eye or eyebrow motion under eye-video control, while head pose remains tied to the geometric bottleneck.

4. Motion-aware encoding, training signals, and evaluation in talking-head synthesis

DisCoHead provides two mechanisms for aligning source appearance to driver pose (Hwang et al., 2023). In the dense-motion variant, the Motion-Aware Encoder jointly encodes appearance, computes a dense motion field, and produces a pose-aligned feature map. Identity flow $\mathbf{d}\in\mathbb{S}^2$9 and coarse flow $e\in\mathbb{R}^{256}$0 are combined by a motion mask $e\in\mathbb{R}^{256}$1,

$e\in\mathbb{R}^{256}$2

then used to warp features before confidence weighting. In the neural-mix variant, explicit warping is removed and the encoder learns to mix features from the original and geometrically transformed inputs implicitly. The reported empirical result is that neural mix is almost on a par with the dense-motion variant, suggesting that explicit flow is not strictly necessary for strong performance.

Supervision is deliberately simple. The total loss is

$e\in\mathbb{R}^{256}$3

where $e\in\mathbb{R}^{256}$4 and the perceptual term is computed on VGG-19 features. The paper explicitly does not mention adversarial loss, landmark-specific losses, regularization on affine or TPS parameters, or audio-visual synchronization losses such as SyncNet-style objectives. The claim is therefore architectural: disentanglement is obtained primarily from the bottlenecked flow of pose information and the separate expression-conditioning channels.

Training and evaluation use the Obama, GRID, and KoEBA datasets, all split 80% train and 20% test, with faces cropped to $e\in\mathbb{R}^{256}$5. Metrics include PSNR, SSIM, FID, LPIPS, Average Keypoint Distance (AKD), and Average Euclidean Distance (AED) from a face-recognition embedding. An example reported for Obama is DisCoHead-Affine-DM at PSNR 28.39, SSIM 0.904, FID 0.618, LPIPS 0.051, AKD 0.915, and AED 0.031. The paper further reports that both affine and TPS variants perform well, and that dense motion and neural mix differ only slightly on several datasets. Qualitatively, the cited figures show independent swapping of head pose, lip movements, and eye blinks. The discussion identifies difficulty with extreme poses, dependence on the training distribution, and the practical requirement of an additional eye-region driving video for full control; it also notes the broader deepfake and misinformation risk characteristic of high-quality talking-head systems.

5. Attention heads as explicit interaction groups

In Transformer LLMs, the “head” is a parallel attention branch rather than a rendered object. Multi-head Explicit Attention (MEA) starts from the observation that standard multi-head attention computes heads independently and that inter-head interaction can enhance attention performance (Peng et al., 27 Jan 2026). MEA introduces a Head-level Linear Composition (HLC) module acting purely along the head dimension. For a tensor $e\in\mathbb{R}^{256}$6 and learnable mixing matrix $e\in\mathbb{R}^{256}$7,

$e\in\mathbb{R}^{256}$8

MEA applies HLC separately to keys and values,

$e\in\mathbb{R}^{256}$9

and adds a head-level GroupNorm layer on the attention outputs.

The paper’s main empirical claim is optimization robustness. In from-scratch pretraining on a 1B-parameter LLaMA3.2-like architecture, MEA remains stable up to a learning rate of $\mathbf{c}\in\mathbb{R}^3$0, whereas the compared variants become unstable beyond $\mathbf{c}\in\mathbb{R}^3$1. In 500B-token training, MEA reaches the best final test loss among the compared methods. On PIQA, OBQA, WinoGrande, HellaSwag, ARC-e, and ARC-c, it reports an average accuracy of 46.39, compared with 45.88 for the baseline Transformer, 45.67 for Transformer plus GroupNorm, and 46.36 for Differential Transformer.

MEA is also used for parameter efficiency through low-rank “virtual heads.” By reducing the number of actual key-value heads and reconstructing the original head set with HLC, the method enables KV-cache compression. The reported practical result is a 50% reduction in KV-cache memory usage with negligible performance loss on knowledge-intensive and scientific reasoning tasks and a 3.59% accuracy drop on Olympiad-level mathematical benchmarks. In the detailed compression experiment, full compression with recovery and continued pretraining yields knowledge average 64.41 versus 65.81 for the uncompressed continued-pretraining baseline, science average 48.79 versus 49.76, math average 46.89 versus 50.48, and total average 52.36 versus 54.39.

6. Shared design motifs and open problems

Across these otherwise distinct domains, a recurring pattern is the use of explicit intermediate structure to prevent uncontrolled entanglement. In HQ3DAvatar, a learned canonical space and deformation field separate geometry from expression-dependent appearance, and temporal correspondences are enforced in canonical coordinates rather than only in image space (Teotia et al., 2023). In DisCoHead, a single affine or TPS transformation acts as the sole channel for head-pose information, while audio and eye-region signals modulate expression through separate pathways (Hwang et al., 2023). In MEA, head-level linear composition and head-level GroupNorm make inter-head dependencies explicit rather than leaving them to the output projection alone (Peng et al., 27 Jan 2026). This suggests a broader design principle: when the target factorization is known in advance, architectural bottlenecks and structured recombination can substitute for heavier supervision.

The limitations are correspondingly domain-specific. For 3D avatars, the open issues are person-specific training, controlled-light capture, and difficult occlusion or topology changes such as tongue emergence. For talking heads, the main constraints are performance under extreme poses, reliance on the training distribution, and the need for an eye-region driving stream for the full three-way control of pose, mouth, and upper-face expression. For inter-head attention, the remaining questions concern head interpretability, rank selection for virtual heads, and the greater sensitivity of hard mathematical reasoning to KV compression.

A common misconception is that template-free or unsupervised formulations eliminate structural bias. The papers indicate a narrower conclusion. HQ3DAvatar is template-free but still relies on a carefully engineered canonical warp, multiresolution hash grids, and flow-based regularization. DisCoHead is unsupervised with respect to pose labels and still constrains pose through a geometric bottleneck. MEA improves over prior inter-head methods only when HLC is coupled with head-level GroupNorm; the paper reports that HLC alone tends to degenerate to baseline behavior. High controllability, whether for digital heads or attention heads, therefore appears to depend less on removing structure than on choosing the appropriate structure and making its role explicit.