HumanMaterial: Canonical Human Appearance

• HumanMaterial is a framework that represents human appearance as a canonical asset integrating body geometry, clothing structure, texture, and reflectance.
• It uses diverse representations such as UV texture maps, physically based rendering tuples, and layered models to facilitate relighting, editing, and animation.
• Methodologies combine human-specific priors, inverse rendering, and diffusion-based generative techniques to manage occlusions, ambiguity, and sparse visual inputs.

HumanMaterial can be understood as the representation, estimation, synthesis, and evaluation of human appearance as a coupled problem of body geometry, clothing structure, texture, and reflectance. In recent work, the target asset is no longer only a pose vector or a body mesh, but a canonical, editable, and often relightable human representation: a full UV texture on an SMPL body, a tuple of physically based rendering maps, a triangle mesh with explicit material textures, or a physically layered body-and-garment model. The central technical difficulty is that observed RGB appearance entangles illumination, geometry, and reflectance, while human-specific factors such as articulation, self-occlusion, loose clothing, and multi-material composition further amplify ambiguity (Casas et al., 2023, Jiang et al., 15 Jun 2026, Chen et al., 2024, Wang et al., 2024).

1. Definition and problem structure

A common formulation of HumanMaterial treats a human as a canonical surface or volumetric object endowed with appearance parameters that remain stable under pose and viewpoint changes. In HAFMat, the task is written as learning a mapping

$\mathcal{F}: I \mapsto (N, D, R, S, SSS),$

where the outputs are normal, diffuse albedo, roughness, specular albedo, and subsurface scattering maps from a single human image. SMPLitex instead models appearance as a complete UV texture map on an SMPL body and reconstructs the full texture from visible image evidence. HR Human pushes the endpoint further toward production assets by recovering a triangular mesh together with explicit physically based material textures from monocular video, while HumanCoser makes the body and clothing separate layers rather than a single coupled surface (Jiang et al., 15 Jun 2026, Casas et al., 2023, Chen et al., 2024, Wang et al., 2024).

This shared structure implies that HumanMaterial is not identical to classical human mesh recovery. Mesh recovery methods estimate body shape and pose; HumanMaterial methods seek a persistent appearance model that supports relighting, editing, synthesis, animation, or reuse. A plausible implication is that the field is defined less by a single representation than by the insistence on canonicalization: visible image evidence is converted into a body-centric description that can survive viewpoint change, pose change, and rendering change.

2. Canonical representations and priors

Canonicalization is the main technical device that makes HumanMaterial tractable. Recent systems differ primarily in what they choose as the canonical appearance carrier and which priors they inject.

Canonical unit	Technical role	Representative system
UV texture map on an SMPL body	complete appearance completion and manipulation	SMPLitex (Casas et al., 2023)
PBR tuple $(N,D,R,S,SSS)$	intrinsic material estimation and relighting	HAFMat (Jiang et al., 15 Jun 2026)
Triangle mesh with $(k_d,r,m)$ UV textures	explicit editable avatar asset	HR Human (Chen et al., 2024)
Separate body and clothing implicit layers	reusable garments and layered animation	HumanCoser (Wang et al., 2024)

In SMPLitex, dense pixel-to-surface correspondences are projected into UV space to form a partial texture map

$$\mathbf{u}_{\text{part}} = \Pi(\mathbf{x}, \mathbf{d} \odot \mathbf{s}),$$

where the correspondence map is masked by a silhouette before projection. Because the output is a single canonical surface representation rather than a posed image, the method argues that multi-view consistency is enforced implicitly (Casas et al., 2023).

HAFMat adopts a different canonical target: five material-related maps. Its guidance set

$$\mathcal{G} = \{I,\; G_{\text{pnor}},\; G_{\text{dp}},\; \mathcal{P}_{HM},\; \mathcal{P}_{AFHR}\}$$

contains the input image, an SMPL-derived normal map, a DensePose map, and prior material predictions from HM and AFHR, for 12 guidance components in total. The architecture assumes these cues are heterogeneous: some are texture-dominant, others semantic-dominant, so they should not be fused uniformly across the decoder (Jiang et al., 15 Jun 2026).

Normals and body models also recur as geometric priors outside explicit PBR estimation. HumanNorm fine-tunes diffusion models on normal maps and conditions RGB synthesis on normals, treating surface orientation as the bridge between text and geometry-aware appearance. HumanCoser uses SMPL skeleton conditioning and an SMPL-driven implicit field deformation network to fit reusable clothing to different bodies. This suggests that HumanMaterial systems rarely operate without a human prior; even when the final output is richly detailed, the scaffold is usually SMPL, SMPL-X, DensePose, or a related articulated parameterization (Huang et al., 2023, Wang et al., 2024).

3. Estimation from sparse visual evidence

A major branch of HumanMaterial research reconstructs appearance from severely incomplete observations: a single RGB image, a monocular video, or a small multi-view image set. The dominant strategy is to combine a human-specific correspondence or structure prior with a strong appearance prior.

SMPLitex follows a two-stage procedure. First, it fine-tunes a latent diffusion model to generate full UV texture maps of SMPL humans using only 10 UV texture maps and 1,500 iterations. Second, given an input image, it estimates dense pixel-to-surface correspondences, projects visible pixels into UV space, and conditions the diffusion model on this partial UV map to inpaint missing regions. At inference, it uses 50 denoising steps and classifier-free guidance $= 2.0$. The method is explicitly framed as a 2D-to-3D appearance completion system: visible body regions constrain identity and clothing, while the diffusion prior hallucinates occluded areas in a globally plausible way (Casas et al., 2023).

HAFMat addresses the same sparsity from an inverse-rendering perspective rather than a texture-completion perspective. Its Multi-layer Adaptive Feature Fusion Mechanism concatenates guidance and decoder features at stage $s$,

$f_{cat}^s = [f_g^s ; f_d^s],$

predicts spatially varying fusion weights with a lightweight network,

$$[w_g^s, w_d^s] = \mathrm{Softmax}(\phi^s(f_{cat}^s)),$$

and outputs

$f_{out}^s = w_g^s \odot f_g^s + w_d^s \odot f_d^s.$

Fusion is injected only at shallow, medium, and deep stages, with Transformer-based fusion at the deep stage. Supervision combines direct material losses with a relighting loss over 37 point-light conditions using a physics-based PBR shader with the Disney BSDF model (Jiang et al., 15 Jun 2026).

Loose clothing produces a distinct failure mode for appearance-aware human recovery because outer contours no longer approximate body shape. ClothHMR addresses this by placing a Clothing Tailoring module before mesh recovery. The tailoring stage predicts a smooth body semantic mask, a sharper body edge map, and a cloth-cutting output, which is multiplied with the original image to produce a tailored body image. Mesh recovery then iteratively aligns rendered joints, depth, and silhouette of an SMPL mesh to cues from a single foundational human visual model over 40 optimization iterations. The method’s stated motivation is that tailoring clothing to fit the body silhouette reduces the adverse impact of clothing on downstream recovery, while a single foundational model yields more internally consistent intermediate cues than separate networks (Gao et al., 19 Dec 2025).

Multi-view and video settings do not eliminate ambiguity; they merely change its form. FastHuman reconstructs a clothed human from a small set of posed multi-view images using an oriented point cloud, mesh-based patch warping, and a Lambertian shape-from-shading stage with third-order spherical harmonics. It reports about 5 minutes for NCC-based optimization, about 1 minute for shading refinement, and total runtime around 6 minutes on its synthetic benchmark. HR Human starts from monocular video, learns a deformable neural implicit surface, renders 50 virtual viewpoints around the person, extracts a mesh in a well-stretched canonical pose, applies a normal-direction bias correction to the coarse mesh, and then optimizes explicit PBR textures on the resulting mesh (Lin et al., 2022, Chen et al., 2024).

4. Generative completion, synthesis, and layered clothing

Another branch of HumanMaterial research is explicitly generative. Here the problem is not only to recover a human from evidence, but to synthesize or manipulate plausible human appearance under text, image, or semantic control.

SMPLitex uses latent diffusion for UV-space human appearance because earlier texture estimators are described as producing blurry, low-detail UV maps with limited latent spaces. Its training objective is the standard latent diffusion noise-prediction loss,

$(N,D,R,S,SSS)$0

adapted to human UV texture maps. Because the system is generative rather than a deterministic regressor, it supports text-driven synthesis, image-based editing and manipulation, and arbitrary texture generation for SMPL-based animation pipelines (Casas et al., 2023).

HumanNorm argues that generic text-to-image diffusion priors do not understand 3D human geometry and therefore lead to Janus artifacts, smooth or blobby surfaces, distorted body proportions, fake wrinkles, and cartoon-like appearance. It replaces RGB-centric priors with a normal-adapted diffusion model for geometry and a normal-aligned diffusion model for texture. View-dependent text such as “front view” or “back view” and body-aware text such as “upper body” or “full body” are used to reduce ambiguity. The method also introduces progressive positional encoding, progressive SDF loss, increasing DMTET resolution from $(N,D,R,S,SSS)$1 to $(N,D,R,S,SSS)$2 at 3,000 iterations and to $(N,D,R,S,SSS)$3 at 6,000 iterations, and a multi-step SDS refinement for texture. On its reported benchmark it achieves FID $(N,D,R,S,SSS)$4 and CLIP Score $(N,D,R,S,SSS)$5 (Huang et al., 2023).

HumanCoser generalizes generative human appearance to physically layered dressing. Instead of generating a single fused surface, it uses a layer-wise dressed human representation in which the inner body and each clothing item are separate implicit fields. The Dual-Representation Decoupling framework supervises each clothing layer both as clothing alone and as clothing composed with earlier layers, while a multi-layer fusion volumetric rendering rule chooses the strongest relevant layer contribution along each ray. An SMPL-driven implicit deformation network then adapts clothing to different body shapes. The system is designed for virtual try-on, clothing transfer, and layered human animation, and reports FID $(N,D,R,S,SSS)$6 and CLIP Score $(N,D,R,S,SSS)$7 in its comparison table (Wang et al., 2024).

A recurring controversy concerns whether diffusion-based image realism implies physically meaningful materials. MaterialSeg3D, although not human-specific, gives a direct critique of SDS-based 3D generation: illumination, shadows, specular highlights, and material appearance are entangled in 2D priors, so direct optimization tends to produce spurious correlated material maps that fail under relighting. The proposed remedy is to infer semantic material categories first and then map them to PBR parameters such as metallic and roughness. A plausible implication for HumanMaterial is that high-quality human avatars may require an explicit separation between “looks realistic in training views” and “encodes intrinsic human reflectance” (Li et al., 2024).

5. Datasets, benchmarks, and evaluation protocols

HumanMaterial research depends heavily on evaluation design because the target object combines articulated motion, geometry, and appearance. Recent datasets differ along three axes: geometric fidelity, subject diversity, and annotation richness.

VolHuMe is a high-resolution 4D human dataset recorded with 64 RGB cameras at $(N,D,R,S,SSS)$8 resolution and 32 depth cameras at $(N,D,R,S,SSS)$9, synchronized at 25 fps in a close-range volumetric studio. It contains captures of 104 adult subjects, each about one minute long, for about 156,000 annotated frames. The annotations include multi-view RGB and depth images, high-resolution textured meshes, dense point clouds, SMPL-X fittings, rigged meshes, garment segmentation, detailed hand geometry, and detailed facial geometry via FLAME fitting. The reported final SMPL-X registration quality is 4.73 mm skin error and 20.05% penetrating clothing error. A controlled comparison in the paper reports 926,774 visible reprojected mesh pixels for a 4K full-body image versus 1,582,839 pixels for the close-up rig, supporting the claim that close-range partial views can provide superior effective sampling (Martinelli et al., 22 Jun 2026).

At the opposite end of the realism spectrum, the synthetic RGB-D human model dataset of seven motion sequences provides clean articulated ground truth for skeleton-driven reconstruction. Each frame includes RGB and depth images, ground-truth mesh geometry, and ground-truth skeleton joints, together with a single synthetic human mesh of 17,021 vertices and 31,492 faces driven by a 14-bone skeleton. Sequence names include Jump Balance, Punch Strike, Boxing, Sword Play, Exercise, Kick Ball, and Direct Traffic. Because skeleton joints are given in camera coordinates while meshes are in world coordinates, evaluation pipelines must handle coordinate transforms correctly (Elanattil et al., 2019).

Task-specific appearance benchmarks are equally important. SMPLitex is evaluated on DeepFashion-MultiModal, Market-1501, and THUman2.0. On Market-1501, it reports SSIM $(k_d,r,m)$0 and LPIPS $(k_d,r,m)$1, compared with TexFormer’s SSIM $(k_d,r,m)$2 and LPIPS $(k_d,r,m)$3. On THUman2.0, under the paper’s cross-view rendering protocol based on 3D scan ground truth, it reports SSIM $(k_d,r,m)$4 and LPIPS $(k_d,r,m)$5, compared with TexFormer’s 0.8761 and 0.1223. The same work also introduces a generated SMPLitex dataset of 100 high-quality UV texture maps sampled from text prompts describing outfit combinations, costumes, sports apparel, job titles, and facial characteristics (Casas et al., 2023).

OpenHumanBRDF plays the analogous role for single-image human PBR estimation. It contains 147 subjects, split into 127 for training and 20 for testing, with $(k_d,r,m)$6 images, foreground masks, ground-truth material maps, and five relighting results under HDR environment maps. HAFMat reports total PSNR $(k_d,r,m)$7, exceeding HM’s 27.2, and total relighting PSNR $(k_d,r,m)$8, exceeding HM’s 23.2 (Jiang et al., 15 Jun 2026).

Additional benchmarks expose clothing and efficiency trade-offs. ClothHMR trains on Cloth4D and THuman2.0 and reports 0.846 mIoU on Cloth4D, plus 46.9 MPJPE, 35.6 PA-MPJPE, and 63.6 MVPE on THuman2.0. FastHuman builds a synthetic benchmark from 40 RenderPeople scans rendered into 19 views at $(k_d,r,m)$9, where it reports normal error $$\mathbf{u}_{\text{part}} = \Pi(\mathbf{x}, \mathbf{d} \odot \mathbf{s}),$$0, Chamfer-$$\mathbf{u}_{\text{part}} = \Pi(\mathbf{x}, \mathbf{d} \odot \mathbf{s}),$$1, PSNR $$\mathbf{u}_{\text{part}} = \Pi(\mathbf{x}, \mathbf{d} \odot \mathbf{s}),$$2, optimization time $$\mathbf{u}_{\text{part}} = \Pi(\mathbf{x}, \mathbf{d} \odot \mathbf{s}),$$3 min, and rendering time $$\mathbf{u}_{\text{part}} = \Pi(\mathbf{x}, \mathbf{d} \odot \mathbf{s}),$$4 s (Gao et al., 19 Dec 2025, Lin et al., 2022).

6. Applications, failure modes, and adjacent debates

The practical endpoints of HumanMaterial are explicit and varied. SMPLitex is presented as a drop-in replacement for textures in SMPL pipelines and supports editing, synthesis, and manipulation. HR Human targets standard graphics engines by exporting a triangular mesh with editable albedo, roughness, and metalness textures. HumanCoser is designed for virtual try-on and layered human animation. HAFMat directly addresses relighting and digital human rendering by producing physically based maps from a single image. ClothHMR is motivated partly by online fashion and shopping, and its supplementary material describes a web application with 360-degree body visualization and virtual fitting (Casas et al., 2023, Chen et al., 2024, Wang et al., 2024, Jiang et al., 15 Jun 2026, Gao et al., 19 Dec 2025).

Despite rapid progress, failure modes remain highly structured. SMPLitex degrades when subjects are heavily occluded or correspondences fail, and overly general or non-human text prompts can yield deformed faces, missing limbs, or implausible body features. HumanNorm still requires a rigged human skeleton for animation, may contain undesired shading in the generated texture, and explicitly identifies PBR material estimation and relighting as future work. HAFMat notes that strong reflections or shadows can still contaminate diffuse albedo, and that self-occlusion shadows estimated by SMPL ray tracing are biased because SMPL geometry does not perfectly match real bodies. FastHuman depends on good masks and camera calibration, assumes predominantly Lambertian reflectance in its shading stage, and can fail in textureless regions or under complex lighting. ClothHMR acknowledges failure on unconventional poses and extreme obesity because the underlying SMPL model has limited expressivity (Casas et al., 2023, Huang et al., 2023, Jiang et al., 15 Jun 2026, Lin et al., 2022, Gao et al., 19 Dec 2025).

VolHuMe sharpens a broader methodological critique. Its benchmarks show that vanilla NeRF and Gaussian splatting methods do not converge well on raw close-range sparse-view data unless the background is segmented and replaced by a feature-rich background, indicating that current reconstruction pipelines remain fragile under realistic volumetric capture. Even strong methods still oversmooth or miss the finest geometric and textural details of the ground-truth meshes (Martinelli et al., 22 Jun 2026).

Adjacent research clarifies the conceptual stakes. MaterialSeg3D argues that semantic material understanding is necessary if 3D assets are to be relightable rather than merely photorealistic in training views. Mesh2Fab, in a fabrication setting, formalizes that shape and material are not independent: wood tends to favor thicker parts and near-right-angle joints, whereas metal allows thinner and curved forms. MateRobot shows, from a human-centered robotics perspective, that material recognition has operational value: its MateViT model reports 40.2% mIoU on COCOStuff-10K and 51.1% on DMS, while the wearable system attains a NASA-Task Load Index score of 28 in field testing. Although these systems are not digital-human avatar methods, they suggest that HumanMaterial is part of a wider shift from treating material as a purely visual afterthought to treating it as a first-class semantic and physical property (Li et al., 2024, Yang et al., 2014, Zheng et al., 2023).