PALM-Net: Physically Based Hand Avatar Reconstruction

• The paper's primary contribution is an integrated framework that combines inverse rendering, continuous implicit 3D surface parameterization, and Disney-style BRDFs to achieve personalized hand avatars.
• It leverages a diverse dataset of 13,000 hand scans and 90,000 multi-view images from 263 subjects to capture rich intra- and inter-subject variability in geometry, skin tone, and appearance.
• The method integrates differentiable volume rendering with specialized neural networks to deliver robust novel view synthesis and improved performance metrics such as PSNR, SSIM, and LPIPS.

PALM-Net is a multi-subject, physically based, inverse-rendering–based hand prior that enables the construction of personalized, relightable 3D hand avatars from single RGB images. Developed as the baseline method in the PALM dataset initiative, PALM-Net is trained on 13,000 high-quality hand scans and 90,000 multi-view images from 263 subjects, capturing rich intra- and inter-subject variability in geometry, skin tone, and appearance. Its primary contribution is an integrated framework that leverages continuous implicit 3D surface parameterization, spatially varying reflectance, and a global scene illumination model, optimized end-to-end via differentiable volume rendering and physically based Disney-style BRDFs.

1. Problem Formulation and Objectives

PALM-Net addresses the task of image-based hand avatar personalization under the following problem setting: the input is a single RGB image $I$ of a right hand, captured in an arbitrary pose and under complex, largely unknown illumination. The method assumes access to a predicted 3D hand pose $\theta$ and shape $\beta$ for the frame, typically provided by off-the-shelf 3D keypoint/pose estimators, such as those used in InterHand2.6M.

The outputs of PALM-Net are:

• A relightable, personalized 3D hand avatar, comprising:
  • Geometry: a continuous implicit surface representation (SDF-to-opacity field) supporting articulation to arbitrary hand poses $\theta'$,
  • Material properties: spatially varying albedo $\alpha(x)$, roughness $r(x)$, and metallicity $m(x)$,
  • Scene illumination: an explicit environment map $L_{\text{env}}(d)$, parameterized by Spherical Gaussians for novel-view relighting.
• Novel renderings of the avatar under new poses and/or environment maps.

This enables application to single-image hand avatar personalization, including relighting and free-form gesture animation.

2. Hand Geometry and Material Parameterization

2.1 Canonicalization via MANO and SNARF

Hand geometry is represented using the MANO parametric model, $\Theta = \{\theta, \beta, p\}$, where $\theta \in \mathbb{R}^{45}$ denotes joint-angle pose, $\beta \in \mathbb{R}^{10}$ are the shape PCA coefficients, and $p\in\mathbb{R}^6$ encodes global rigid transformation. SNARF [Chen et al. ICCV’21] is utilized to invert linear blend skinning, mapping each deformed point $x_d$ to a common canonical space $x_c$ by solving: $$x_c = \arg\min_{x} \bigg\lVert \sum_{i=1}^{n_b} w_i(x) B_i x - x_d \bigg\rVert_2^2$$ where $w_i(x)$ are skinning weights and $B_i$ bone transforms.

2.2 Implicit Geometry Network

A continuous, canonical surface representation is learned via an SDF-to-opacity MLP $f_g$: $$f_g: (x_c, \theta, \beta, \phi) \mapsto (\sigma_t(x_c), z(x_c))$$ Here, $x_c$ is the SNARF-canonicalized point, $\phi \in \mathbb{R}^{d_s}$ is a learned subject-specific latent geometry code ($d_s \approx 32$), $\sigma_t(x_c)$ is the local volume density (opacity, via Laplace-CDF mapping), and $z(x_c)$ is a learned geometry feature vector forwarded to appearance sub-networks.

2.3 Neural Radiance and Material Fields

The radiance field network $f_{rf}$ models view-dependent surface emission given geometry, normal, and an appearance code $\psi$: $$f_{rf}: (x_c, z, \mathrm{ref}(d, n), n, \theta, \psi) \rightarrow L(x_c, d)$$

• Surface normal $n = \nabla_x \mathrm{SDF}(x_c)$,
• Reflected direction $\mathrm{ref}(d, n) = d - 2(d \cdot n)n$,
• Appearance code $\psi \in \mathbb{R}^{d_s}$.

The material property network $f_m$ outputs: $$f_m: (x_c, z, \theta, \psi) \rightarrow (\alpha(x_c), r(x_c), m(x_c))$$ where $\alpha$ is spatially varying, per-axis albedo, $r$ the roughness scalar, and $m$ metallicity.

For both geometry and appearance, each subject $i$ receives distinct latent codes $\phi_i$ (shape) and $\psi_i$ (appearance), learned end-to-end.

3. Network Architecture

Each network ($f_g$, $f_{rf}$, $f_m$) is implemented as an MLP, leveraging hash-grid positional encodings [Müller et al. TOG’22] of $x_c$. Typical configurations use 4–6 layers of 128–256 channels per layer. Conditioning is achieved by concatenating the hash-encoded $x_c$, subject latent codes, MANO pose/shape embeddings, and, where appropriate, geometry features and view/normal directions. The overall connectivity is as follows:

Network	Inputs	Outputs
$f_g$	$x_c$ (hash-encoded), $\phi$, MANO $\theta,\beta$	$\sigma_t(x_c)$, $z(x_c)$
$f_{rf}$	$x_c$, $z$, $n$, $\mathrm{ref}(d,n)$, $\theta$, $\psi$	$L(x_c, d)$ (RGB radiance)
$f_m$	$x_c$, $z$, $\theta$, $\psi$	$\alpha$, $r$, $m$

This architecture enables the model to capture both global inter-subject variation and personalized detail through subject-specific codes while maintaining parameter efficiency via shared weights across all subjects.

4. Physically Based Inverse Rendering

4.1 Differentiable Volume Rendering

PALM-Net utilizes NeRF-style differentiable volume rendering: $$C_{rf}(r) = \int_{t_n}^{t_f} T(t_n, t)\,\sigma_t(r(t))\,L(r(t),d)\,dt$$ where the transmittance $$T(t_n, t) = \exp\left(-\int_{t_n}^t \sigma_t(r(s))ds\right)$$ encodes accumulated opacity. In practical implementation, this integral is approximated by standard quadrature.

4.2 Physically-Based Rendering with Disney-BRDF

The radiance $L$ in the above is replaced, during training, by a physically based integral using the Disney BRDF: $$C_{pbr}(r) \approx \sum_{i=1}^M w^{(i)} \,\mathrm{BRDF}(d, \bar d^{(i)}, \alpha, r, m, n)\,\frac{L_i(r(\bar t^{(i)}), \bar d^{(i)})}{pdf(\bar d^{(i)})}$$ where weights $w^{(i)}$ are computed from opacity, $\mathrm{BRDF}$ applies the Disney model, $L_i$ incorporates outgoing and environment radiance, and $\bar d^{(i)}$ are sampled incoming directions. The environment map is modeled as a sum of $G=8$ learnable Spherical Gaussians. All integrals are differentiable, allowing backpropagation through illumination, geometry, and material networks.

5. Loss Functions and Training Objectives

Supervision is provided by registered 3dMD scan renderings, including ground-truth color $\hat C(r)$, normal $\hat{\mathcal N}(r)$, and segmentation $\hat{\mathcal S}(r)$. The loss is a weighted sum: $$\mathcal{L} = \mathcal{L}_{rf} + \lambda_{pbr} \mathcal{L}_{pbr} + \lambda_{seg} \mathcal{L}_{seg} + \lambda_{normal} \mathcal{L}_{normal} + \lambda_{eik} \mathcal{L}_{eik} + \lambda_{LAP} \mathcal{L}_{LAP} + \lambda_{latent} \mathcal{L}_{latent} + \lambda_{LPIPS} \mathcal{L}_{LPIPS}$$

• $\mathcal{L}_{rf}, \mathcal{L}_{pbr}$: Photometric $\ell_1$ losses in rendered color.
• $\mathcal{L}_{normal}$: $\ell_2$ supervision on predicted vs. scan normals.
• $\mathcal{L}_{seg}$: Foreground binary cross-entropy loss.
• $\mathcal{L}_{eik}$: Eikonal regularizer $(\|\nabla_x \mathrm{SDF}(x)\|_2 - 1)^2$, enforcing SDF validity.
• $\mathcal{L}_{LAP}$: Laplacian smoothness penalty to encourage local surface consistency.
• $\mathcal{L}_{latent}$: Regularization of subject latent code norms.
• $\mathcal{L}_{LPIPS}$: Patch-based perceptual loss for high-frequency appearance details.

$\lambda$ coefficients are determined empirically; $\lambda_{seg}$ is decayed progressively during training.

6. Training Protocol and Dataset

Training uses the PALM dataset, which provides 13,000 hand scans from 263 subjects, with 8–12 synchronized high-resolution (2448×2048) RGB images per instance, supporting multi-view supervision. A single multi-subject model is trained across all subjects. Latent codes ($\phi_i$, $\psi_i$) are initialized at zero and learned jointly with model weights. The global scene environment is parameterized by $G=8$ Spherical Gaussians. Optimization uses Adam (learning rate $5 \cdot 10^{-4}$) with exponential decay every 50k iterations, across approximately 200k total iterations. Each mini-batch samples roughly 1024 rays from randomly chosen images/cameras, ensuring subject and pose diversity. Data augmentations include random gamma, color jitter, and patch cropping for perceptual loss.

7. Evaluation, Results, and Ablation Analysis

Single-image personalization is evaluated by fitting subject codes and scene environment parameters to a target image (fixing network weights). Performance is reported in peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and LPIPS:

	PSNR↑	SSIM↑	LPIPS↓
InterHand2.6M
Handy	7.50	0.69	0.24
HARP	9.89	0.78	0.16
UHM	10.08	0.76	0.19
Ours (PALM-Net)	12.01	0.84	0.15
HARP relit
Handy	12.48	0.76	0.32
HARP	11.93	0.69	0.37
UHM	12.30	0.74	0.31
Ours (PALM-Net)	13.39	0.78	0.35

Qualitative results indicate faithful geometry, albedo estimation, and accurate relighting under both real-world and synthetic variations.

Ablation studies assess the impact of normal supervision and adaptive environment fitting:

• Training with normal supervision (3dMD ground-truth normals) reduces "pepper" and "floater" artifacts and improves fidelity (e.g., PSNR improves from 11.97 without normals to 12.01 with normals).
• Optimizing environment Spherical Gaussian parameters during personalization (as opposed to holding them fixed) leads to better fit and reconstruction accuracy (Table 6).

8. Significance and Applications

PALM-Net establishes a strong multi-subject hand prior that synthesizes geometry, reflectance, and illumination in a unified, physically based neural rendering framework. This supports a range of applications such as creation of high-quality, relightable hand avatars from single images, forensic hand modeling, gesture analysis, and personalization of hand models for virtual/augmented reality. The method demonstrates robust generalization across real and synthetic conditions, validated through both numerical and qualitative results. The PALM dataset, together with PALM-Net, addresses prior limitations in subject diversity and physical accuracy in hand avatar reconstruction, providing a scalable resource for further research in photorealistic hand modeling and human digital twin approaches.