---
title: 'ClothHMR: 3D Mesh Recovery Under Clothing'
url: https://www.emergentmind.com/topics/clothhmr
type: topic
---

# ClothHMR: 3D Mesh Recovery Under Clothing

Searching arXiv for ClothHMR and closely related papers mentioned in the provided material.
First, I’ll look up the ClothHMR paper itself and then the specific baseline/foundation papers explicitly named in the data.
ClothHMR is a single-image 3D human mesh recovery framework for humans in diverse clothing, with particular emphasis on loose garments, where conventional human mesh recovery pipelines degrade in body-shape and pose estimation. It is built around two stated insights: first, that tailoring clothing to better match the underlying body silhouette can mitigate the adverse effect of garments on reconstruction; second, that intermediate visual cues extracted from a large foundational human vision model can improve generalization. The framework therefore combines a Clothing Tailoring module with an FHVM-based Mesh Recovering module, and is reported to outperform prior methods on Cloth4D, THuman2.0, EMDB, and 3DPW while also supporting a web-based virtual-try-on and shopping application [2512.17545].

## 1. Problem setting and motivating observations

ClothHMR is situated within 3D human mesh recovery (HMR), where the objective is to infer a parametric 3D human body mesh from a single RGB image. The paper identifies a specific failure mode of existing HMR systems: methods such as HMR, SPIN, PyMAF, and TokenHMR are described as being developed primarily for people in tight clothing and trained on datasets such as Human3.6M, MPI-INF-3DHP, 3DPW, COCO, and LSP, which lack substantial coverage of loose garments [2512.17545].

Under diverse or loose clothing, three concrete failure patterns are emphasized. Existing systems may confuse body silhouette with garment folds, mis-estimate limb lengths and joint positions, and produce implausible body shapes under occlusion. These observations motivate the two central design premises of ClothHMR. The first is that if excess garment regions are effectively “tailored” away, the resulting image becomes closer to the true body silhouette and thus more amenable to robust mesh recovery. The second is that a single foundational human vision model can provide internally consistent 2D and 3D cues—joints, silhouette, and depth—more reliably than assembling several specialized predictors [2512.17545].

A common misconception is to treat this setting as garment reconstruction. ClothHMR is instead formulated as body mesh recovery under clothing: the target is the human mesh, not an explicit garment surface. This is reflected throughout the pipeline, which operates on body masks, SMPL parameters, and rendered pose/depth/silhouette cues rather than clothing geometry.

## 2. End-to-end formulation and pipeline

The architecture consists of two sequential modules: Clothing Tailoring (CT) and FHVM-based Mesh Recovering (MR). Given one RGB image $\mathcal I$, the CT module predicts a body-only mask $\mathcal C_r$ and produces a tailored image
$$
\mathcal B = \mathcal C_r \odot \mathcal I.
$$
This tailored image is then passed to the MR stage [2512.17545].

The MR module begins from an initial SMPL mesh estimate $\mathcal M_0$ provided by an off-the-shelf regressor. In parallel, an FHVM encoder-decoder extracts three intermediate cues from $\mathcal B$: 2D joint heatmaps $\mathcal G_j \in \mathbb R^{H \times W \times N_k}$, a normalized depth map $\mathcal G_d \in [0,1]^{H \times W}$, and a body silhouette $\mathcal G_m \in [0,1]^{H \times W}$. ClothHMR then iteratively optimizes the SMPL parameters $\{\beta,\theta,s,t\}$ so that rendered cues from the current mesh estimate align with these FHVM-derived cues. The final output is a refined mesh $\mathcal M_r$ [2512.17545].

This decomposition is significant because it separates two entangled difficulties. CT attempts to suppress the appearance-level ambiguity introduced by clothing, while MR constrains the body mesh through multi-viewpoint-consistent intermediate representations, albeit inferred from a single image. This suggests that the method is designed to reduce both image-space contamination and model-space drift.

## 3. Clothing Tailoring module

The Clothing Tailoring module contains three subnetworks: body semantic estimation, body edge prediction, and cloth cutting. Its purpose is to predict a body mask that better approximates the unclothed body silhouette despite the presence of loose garments [2512.17545].

In body semantic estimation, an encoder $\mathcal E$ extracts multi-scale features $\{\mathcal E_F^k\}$. A $1\times1$ convolution followed by Sigmoid on $\mathcal E_F^5$ produces a smooth semantic mask $\mathcal C_m \in [0,1]^{H \times W}$. The associated segmentation loss is given as
$$
L_{\text{seg}} = \frac12 \bigl\|\mathcal C_m - G_{16\times}(\alpha_g)\bigr\|_2^2.
$$

In body edge prediction, the image pyramid $\{\mathcal I_{1/2^k}\}$ is fused with $\{\mathcal E_F^k\}$ through Feature Pyramid Fusion, described as concatenation followed by small convolutions, to produce edge feature maps $\mathcal H_F^k$. A final convolution yields an edge map $\mathcal H_r$. The edge alignment loss is a masked $L_1$ term against the ground-truth boundary:
$$
L_{\text{edge}} = \bigl\|\,\mathcal H_r - \alpha_g\bigr\|_{1,\,m_d},
$$
where $m_d$ is a binary mask of the true silhouette boundary obtained by dilation and erosion.

In cloth cutting, decoded features from the edge branch are fused with upsampled body semantic features through Decoded Feature Fusion to predict the final body mask $\mathcal C_r$. The loss is
$$
L_{\text{cut}}
=\|\mathcal C_r-\alpha_g\|_1
\;+\;L_{\text{sim}}(\mathcal C_r,\alpha_g)
\;+\;L_{\text{KL}}(\mathcal C_r,\alpha_g).
$$
The total CT objective is
$$
L_{\text{CT}}
=\lambda_h\,L_{\text{edge}}
\;+\;\lambda_c\,L_{\text{cut}}
\;+\;\lambda_m\,L_{\text{seg}},
$$
with $\lambda_h=\lambda_c=4$ and $\lambda_m=2$ [2512.17545].

The CT stage is central to the method’s interpretation of clothing. It does not estimate garment physics or semantic clothing categories; rather, it predicts a body-conforming mask by combining region-level semantic evidence with boundary-level geometric evidence. A plausible implication is that this design is especially valuable when garment folds and self-occlusions would otherwise dominate the silhouette used by downstream HMR.

## 4. FHVM-based mesh recovering

The FHVM-based Mesh Recovering module uses a foundational human vision model to generate the intermediate signals that drive mesh fitting. Its encoder is a ViT backbone pre-trained by MAE on 300 M+ human images, identified in the paper as Sapiens. Three lightweight decoders share the encoded tokens $F_{\rm out}$: a joint heatmap decoder $\mathcal J$, a depth decoder $\mathcal D$, and a silhouette decoder $\mathcal M$ [2512.17545].

The training losses for these outputs are specified explicitly. For joint heatmaps, the paper uses MSE:
$$
L_{\text{pose}}=\|J_{gt}-\mathcal G_j\|_2^2.
$$
For depth, it uses a scale-invariant $L_2$ objective:
$$
\Delta d=\log D_{gt}-\log\mathcal G_d,\quad
L_{\text{depth}}
=\sqrt{\tfrac1{N_p}\sum\Delta d_i^2
\;-\;\tfrac12\Bigl(\tfrac1{N_p}\sum\Delta d_i\Bigr)^2}.
$$
For silhouette prediction, it uses
$$
L_{\text{mask}}=\|\mathcal G_m - M_{gt}\|_1.
$$

Mesh recovery parameterizes the body through SMPL as $M(\beta,\theta)$ with global scale $s$ and translation $t$. From the current estimate, the method renders 2D joints $\mathcal P_j$, depth $\mathcal P_d$, and silhouette $\mathcal P_m$, then fits parameters by minimizing
$$
L_{\text{MR}}(\beta,\theta,s,t)
=\lambda_d\,\|\mathcal G_d - \mathcal P_d\|_1
+\lambda_m\,\|\mathcal G_m - \mathcal P_m\|_1
+\lambda_j\,\|\mathcal G_j - \mathcal P_j\|_2^2,
$$
with $\lambda_d=\lambda_m=5$ and $\lambda_j=10$. Optimization is performed with a simple gradient-based or Gauss–Newton style solver for 40 iterations [2512.17545].

This stage formalizes the second core insight of the framework. Rather than combining heterogeneous predictors with potentially incompatible inductive biases, ClothHMR relies on one FHVM to produce pose, depth, and silhouette cues in a shared representational space. The paper explicitly argues that these cues are more consistent and higher-fidelity than those obtained by stitching together multiple specialized networks.

## 5. Training protocol and empirical evaluation

The CT module is trained on 80% of Cloth4D, described as 100 K synthetic humans, together with THuman2.0, described as 525 scans, using rendered pairs of clothed and unclothed images. The reported batch size is 16, the initial learning rate is 0.01 with decay by a factor of $0.1$ every 10 epochs, and training runs for 60 epochs on a single V100 for approximately 10 days. FHVM fine-tuning uses the same Cloth4D+THuman2.0 data with supervision from 2D joints, masks, and depth, trained using AdamW on 16 V100 GPUs for 25 days. For inference, MR uses any off-the-shelf regressor for SMPL initialization and runs 40 fitting steps per image [2512.17545].

Evaluation uses Cloth4D with body-mask mIoU, and THuman2.0, EMDB, and 3DPW with MPJPE, PA-MPJPE, and MVPE. The reported results place ClothHMR ahead of SPIN, PyMAF, PyMAF-X, VirtualMarker, MeshPose, and TokenHMR on the listed benchmarks [2512.17545].

| Dataset | Best prior result reported | ClothHMR |
|---|---|---|
| Cloth4D | TokenHMR: 0.799 mIoU | 0.846 mIoU |
| THuman2.0 | TokenHMR: 58.1 / 42.6 / 74.9 | 46.9 / 35.6 / 63.6 |
| EMDB | TokenHMR: 79.8 / 57.2 / 90.8 | 71.7 / 50.6 / 73.6 |
| 3DPW | TokenHMR: 60.8 / 43.7 / 62.8 | 59.9 / 41.9 / 59.2 |

The qualitative analysis states that ClothHMR handles extreme loose coats, dresses, sportswear, and challenging poses better than SPIN, PyMAF, PyMAF-X, VirtualMarker, and TokenHMR. The ablation study further isolates the contribution of each component. Without CT, the method reports 0.768 mIoU on Cloth4D, 68.5 / 48.6 / 85.5 on THuman2.0, and 87.4 / 60.2 / 90.9 on EMDB. Replacing the MR module with TokenHMR while keeping CT yields 0.806 mIoU, 56.1 / 41.4 / 71.6, and 73.3 / 53.7 / 83.8. Removing FHVM yields 0.809 mIoU, 61.4 / 43.8 / 80.6, and 78.7 / 55.3 / 85.7. Full ClothHMR reaches 0.846 mIoU, 46.9 / 35.6 / 63.6, and 71.7 / 50.6 / 73.6 [2512.17545].

The paper summarizes these ablations with three explicit takeaways: CT alone brings +10% mIoU and approximately 25% MPJPE reduction; FHVM cues reduce error by approximately 20% relative to OpenPose+MODNet+HDNet; and the simple iterative MR stage outperforms TokenHMR when given the CT image.

## 6. Applications, limitations, and prospective extensions

ClothHMR is not presented solely as a benchmark model. The paper describes a deployed web-based virtual-try-on and shopping demo in which users upload a selfie in any clothing, the CT-MR pipeline runs server-side, and a 3D avatar representing body and pose is displayed for viewing and fitting with new garments in real time [2512.17545].

This deployment serves two roles. First, it demonstrates that the pipeline can function in real-world usage scenarios rather than only curated benchmark conditions. Second, it highlights the kinds of downstream applications enabled by single-image body mesh recovery under clothing. The paper explicitly mentions future extensions including online fitness, AR/VR content creation, and immersive gaming.

The limitations are also stated directly. ClothHMR still struggles with extremely unconventional poses, exemplified by acrobatics, and with extreme body shapes, exemplified by severe obesity, when these lie outside SMPL’s linear subspace. The proposed future direction is to replace SMPL with non-parametric mesh representations, including learned Free-Form Deformations or implicit fields such as VS, in order to expand expressive capacity [2512.17545].

These limitations clarify the scope of the contribution. ClothHMR improves robustness to clothing-induced ambiguity, but it remains bounded by the representational assumptions of SMPL and by the pose and shape distributions seen during training. A plausible implication is that future progress will depend not only on better visual cues, but also on richer body representations that can absorb out-of-subspace human variation.

Source: https://www.emergentmind.com/topics/clothhmr