ClothHMR: 3D Mesh Recovery Under Clothing

• ClothHMR is a 3D human mesh recovery framework designed to overcome failures in traditional methods by tailoring clothing to better reveal the true body silhouette.
• It combines a Clothing Tailoring module that extracts a refined body mask with an FHVM-based Mesh Recovering module that leverages multi-cue visual signals for accurate SMPL parameter estimation.
• Evaluations show ClothHMR outperforms predecessors on benchmarks like Cloth4D and 3DPW, and supports real-world applications such as web-based virtual try-on, despite limitations with extreme poses and body shapes.

Searching arXiv for ClothHMR and closely related papers mentioned in the provided material. First, I’ll look up the ClothHMR paper itself and then the specific baseline/foundation papers explicitly named in the data. ClothHMR is a single-image 3D human mesh recovery framework for humans in diverse clothing, with particular emphasis on loose garments, where conventional human mesh recovery pipelines degrade in body-shape and pose estimation. It is built around two stated insights: first, that tailoring clothing to better match the underlying body silhouette can mitigate the adverse effect of garments on reconstruction; second, that intermediate visual cues extracted from a large foundational human vision model can improve generalization. The framework therefore combines a Clothing Tailoring module with an FHVM-based Mesh Recovering module, and is reported to outperform prior methods on Cloth4D, THuman2.0, EMDB, and 3DPW while also supporting a web-based virtual-try-on and shopping application (Gao et al., 19 Dec 2025).

1. Problem setting and motivating observations

ClothHMR is situated within 3D human mesh recovery (HMR), where the objective is to infer a parametric 3D human body mesh from a single RGB image. The paper identifies a specific failure mode of existing HMR systems: methods such as HMR, SPIN, PyMAF, and TokenHMR are described as being developed primarily for people in tight clothing and trained on datasets such as Human3.6M, MPI-INF-3DHP, 3DPW, COCO, and LSP, which lack substantial coverage of loose garments (Gao et al., 19 Dec 2025).

Under diverse or loose clothing, three concrete failure patterns are emphasized. Existing systems may confuse body silhouette with garment folds, mis-estimate limb lengths and joint positions, and produce implausible body shapes under occlusion. These observations motivate the two central design premises of ClothHMR. The first is that if excess garment regions are effectively “tailored” away, the resulting image becomes closer to the true body silhouette and thus more amenable to robust mesh recovery. The second is that a single foundational human vision model can provide internally consistent 2D and 3D cues—joints, silhouette, and depth—more reliably than assembling several specialized predictors (Gao et al., 19 Dec 2025).

A common misconception is to treat this setting as garment reconstruction. ClothHMR is instead formulated as body mesh recovery under clothing: the target is the human mesh, not an explicit garment surface. This is reflected throughout the pipeline, which operates on body masks, SMPL parameters, and rendered pose/depth/silhouette cues rather than clothing geometry.

2. End-to-end formulation and pipeline

The architecture consists of two sequential modules: Clothing Tailoring (CT) and FHVM-based Mesh Recovering (MR). Given one RGB image $\mathcal I$, the CT module predicts a body-only mask $\mathcal C_r$ and produces a tailored image

$\mathcal B = \mathcal C_r \odot \mathcal I.$

This tailored image is then passed to the MR stage (Gao et al., 19 Dec 2025).

The MR module begins from an initial SMPL mesh estimate $\mathcal M_0$ provided by an off-the-shelf regressor. In parallel, an FHVM encoder-decoder extracts three intermediate cues from $\mathcal B$: 2D joint heatmaps $\mathcal G_j \in \mathbb R^{H \times W \times N_k}$, a normalized depth map $\mathcal G_d \in [0,1]^{H \times W}$, and a body silhouette $\mathcal G_m \in [0,1]^{H \times W}$. ClothHMR then iteratively optimizes the SMPL parameters $\{\beta,\theta,s,t\}$ so that rendered cues from the current mesh estimate align with these FHVM-derived cues. The final output is a refined mesh $\mathcal M_r$ (Gao et al., 19 Dec 2025).

This decomposition is significant because it separates two entangled difficulties. CT attempts to suppress the appearance-level ambiguity introduced by clothing, while MR constrains the body mesh through multi-viewpoint-consistent intermediate representations, albeit inferred from a single image. This suggests that the method is designed to reduce both image-space contamination and model-space drift.

3. Clothing Tailoring module

The Clothing Tailoring module contains three subnetworks: body semantic estimation, body edge prediction, and cloth cutting. Its purpose is to predict a body mask that better approximates the unclothed body silhouette despite the presence of loose garments (Gao et al., 19 Dec 2025).

In body semantic estimation, an encoder $\mathcal C_r$0 extracts multi-scale features $\mathcal C_r$1. A $\mathcal C_r$2 convolution followed by Sigmoid on $\mathcal C_r$3 produces a smooth semantic mask $\mathcal C_r$4. The associated segmentation loss is given as

$\mathcal C_r$5

In body edge prediction, the image pyramid $\mathcal C_r$6 is fused with $\mathcal C_r$7 through Feature Pyramid Fusion, described as concatenation followed by small convolutions, to produce edge feature maps $\mathcal C_r$8. A final convolution yields an edge map $\mathcal C_r$9. The edge alignment loss is a masked $\mathcal B = \mathcal C_r \odot \mathcal I.$0 term against the ground-truth boundary:

$\mathcal B = \mathcal C_r \odot \mathcal I.$1

where $\mathcal B = \mathcal C_r \odot \mathcal I.$2 is a binary mask of the true silhouette boundary obtained by dilation and erosion.

In cloth cutting, decoded features from the edge branch are fused with upsampled body semantic features through Decoded Feature Fusion to predict the final body mask $\mathcal B = \mathcal C_r \odot \mathcal I.$3. The loss is

$\mathcal B = \mathcal C_r \odot \mathcal I.$4

The total CT objective is

$\mathcal B = \mathcal C_r \odot \mathcal I.$5

with $\mathcal B = \mathcal C_r \odot \mathcal I.$6 and $\mathcal B = \mathcal C_r \odot \mathcal I.$7 (Gao et al., 19 Dec 2025).

The CT stage is central to the method’s interpretation of clothing. It does not estimate garment physics or semantic clothing categories; rather, it predicts a body-conforming mask by combining region-level semantic evidence with boundary-level geometric evidence. A plausible implication is that this design is especially valuable when garment folds and self-occlusions would otherwise dominate the silhouette used by downstream HMR.

4. FHVM-based mesh recovering

The FHVM-based Mesh Recovering module uses a foundational human vision model to generate the intermediate signals that drive mesh fitting. Its encoder is a ViT backbone pre-trained by MAE on 300 M+ human images, identified in the paper as Sapiens. Three lightweight decoders share the encoded tokens $\mathcal B = \mathcal C_r \odot \mathcal I.$8: a joint heatmap decoder $\mathcal B = \mathcal C_r \odot \mathcal I.$9, a depth decoder $\mathcal M_0$0, and a silhouette decoder $\mathcal M_0$1 (Gao et al., 19 Dec 2025).

The training losses for these outputs are specified explicitly. For joint heatmaps, the paper uses MSE:

$\mathcal M_0$2

For depth, it uses a scale-invariant $\mathcal M_0$3 objective:

$\mathcal M_0$4

For silhouette prediction, it uses

$\mathcal M_0$5

Mesh recovery parameterizes the body through SMPL as $\mathcal M_0$6 with global scale $\mathcal M_0$7 and translation $\mathcal M_0$8. From the current estimate, the method renders 2D joints $\mathcal M_0$9, depth $\mathcal B$0, and silhouette $\mathcal B$1, then fits parameters by minimizing

$\mathcal B$2

with $\mathcal B$3 and $\mathcal B$4. Optimization is performed with a simple gradient-based or Gauss–Newton style solver for 40 iterations (Gao et al., 19 Dec 2025).

This stage formalizes the second core insight of the framework. Rather than combining heterogeneous predictors with potentially incompatible inductive biases, ClothHMR relies on one FHVM to produce pose, depth, and silhouette cues in a shared representational space. The paper explicitly argues that these cues are more consistent and higher-fidelity than those obtained by stitching together multiple specialized networks.

5. Training protocol and empirical evaluation

The CT module is trained on 80% of Cloth4D, described as 100 K synthetic humans, together with THuman2.0, described as 525 scans, using rendered pairs of clothed and unclothed images. The reported batch size is 16, the initial learning rate is 0.01 with decay by a factor of $\mathcal B$5 every 10 epochs, and training runs for 60 epochs on a single V100 for approximately 10 days. FHVM fine-tuning uses the same Cloth4D+THuman2.0 data with supervision from 2D joints, masks, and depth, trained using AdamW on 16 V100 GPUs for 25 days. For inference, MR uses any off-the-shelf regressor for SMPL initialization and runs 40 fitting steps per image (Gao et al., 19 Dec 2025).

Evaluation uses Cloth4D with body-mask mIoU, and THuman2.0, EMDB, and 3DPW with MPJPE, PA-MPJPE, and MVPE. The reported results place ClothHMR ahead of SPIN, PyMAF, PyMAF-X, VirtualMarker, MeshPose, and TokenHMR on the listed benchmarks (Gao et al., 19 Dec 2025).

Dataset	Best prior result reported	ClothHMR
Cloth4D	TokenHMR: 0.799 mIoU	0.846 mIoU
THuman2.0	TokenHMR: 58.1 / 42.6 / 74.9	46.9 / 35.6 / 63.6
EMDB	TokenHMR: 79.8 / 57.2 / 90.8	71.7 / 50.6 / 73.6
3DPW	TokenHMR: 60.8 / 43.7 / 62.8	59.9 / 41.9 / 59.2

The qualitative analysis states that ClothHMR handles extreme loose coats, dresses, sportswear, and challenging poses better than SPIN, PyMAF, PyMAF-X, VirtualMarker, and TokenHMR. The ablation study further isolates the contribution of each component. Without CT, the method reports 0.768 mIoU on Cloth4D, 68.5 / 48.6 / 85.5 on THuman2.0, and 87.4 / 60.2 / 90.9 on EMDB. Replacing the MR module with TokenHMR while keeping CT yields 0.806 mIoU, 56.1 / 41.4 / 71.6, and 73.3 / 53.7 / 83.8. Removing FHVM yields 0.809 mIoU, 61.4 / 43.8 / 80.6, and 78.7 / 55.3 / 85.7. Full ClothHMR reaches 0.846 mIoU, 46.9 / 35.6 / 63.6, and 71.7 / 50.6 / 73.6 (Gao et al., 19 Dec 2025).

The paper summarizes these ablations with three explicit takeaways: CT alone brings +10% mIoU and approximately 25% MPJPE reduction; FHVM cues reduce error by approximately 20% relative to OpenPose+MODNet+HDNet; and the simple iterative MR stage outperforms TokenHMR when given the CT image.

6. Applications, limitations, and prospective extensions

ClothHMR is not presented solely as a benchmark model. The paper describes a deployed web-based virtual-try-on and shopping demo in which users upload a selfie in any clothing, the CT-MR pipeline runs server-side, and a 3D avatar representing body and pose is displayed for viewing and fitting with new garments in real time (Gao et al., 19 Dec 2025).

This deployment serves two roles. First, it demonstrates that the pipeline can function in real-world usage scenarios rather than only curated benchmark conditions. Second, it highlights the kinds of downstream applications enabled by single-image body mesh recovery under clothing. The paper explicitly mentions future extensions including online fitness, AR/VR content creation, and immersive gaming.

The limitations are also stated directly. ClothHMR still struggles with extremely unconventional poses, exemplified by acrobatics, and with extreme body shapes, exemplified by severe obesity, when these lie outside SMPL’s linear subspace. The proposed future direction is to replace SMPL with non-parametric mesh representations, including learned Free-Form Deformations or implicit fields such as VS, in order to expand expressive capacity (Gao et al., 19 Dec 2025).

These limitations clarify the scope of the contribution. ClothHMR improves robustness to clothing-induced ambiguity, but it remains bounded by the representational assumptions of SMPL and by the pose and shape distributions seen during training. A plausible implication is that future progress will depend not only on better visual cues, but also on richer body representations that can absorb out-of-subspace human variation.