MonoCloth: Monocular Cloth Capture

• MonoCloth is a monocular cloth-capture framework that reconstructs and animates clothed human avatars by explicitly decoupling body, face, hands, and clothing.
• It employs a two-stage learning scheme with 3D Gaussian representations anchored to SMPL-X and a dedicated CloSim module to model dynamic garment deformations.
• The method achieves high quantitative performance and enables practical applications like clothing transfer while addressing challenges such as occlusion and non-rigid motion.

Searching arXiv for the named paper and closely related monocular cloth-capture work. MonoCloth denotes both a specific 2025 method for monocular avatar reconstruction and a broader research target in monocular cloth capture. In the specific sense, "MonoCloth: Reconstruction and Animation of Cloth-Decoupled Human Avatars from Monocular Videos" reconstructs and animates clothed human avatars from monocular videos by explicitly decoupling body, face, hands, and clothing, and by assigning clothing a dedicated learned spatio-temporal module, CloSim, within an SMPL-X-anchored 3D Gaussian representation (Jin et al., 6 Aug 2025). In the broader sense, the term aligns with a family of methods that attempt to infer garment geometry, appearance, and dynamics from single-view RGB observations despite severe geometric ambiguity, non-rigid motion, self-occlusion, and the mismatch between articulated body motion and history-dependent cloth deformation (Xiang et al., 2020).

1. Research context and lineage

Monocular cloth capture developed through several partially overlapping paradigms. "MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video" addressed temporally coherent garment deformation without a pre-scanned personalized mesh template by building statistical PCA deformation models for T-shirt, short pants and long pants, and by combining silhouette, segmentation, texture, and normal cues in a differentiable-rendering pipeline (Xiang et al., 2020). "Deep Physics-aware Inference of Cloth Deformation for Monocular Human Performance Capture" instead introduced a person-specific, learning-based monocular capture method with separate cloth geometry and a simulation layer embedded into training, claiming the first physics supervision in the weakly supervised deep monocular human performance capture setting (Li et al., 2020).

Subsequent methods diversified the representation. "SCARF (Segmented Clothed Avatar Radiance Field)" used a hybrid model combining a mesh-based body with a segmented neural radiance field, optimized directly from monocular videos without any 3D supervision, and explicitly supported animation, novel-view synthesis, and clothing transfer (Feng et al., 2022). "xCloth: Extracting Template-free Textured 3D Clothes from a Monocular Image" shifted toward template-free textured garment digitization from a single image via layered pixel-aligned depth, semantic, RGB, and normal maps, together with automatic UV atlas generation (Srivastava et al., 2022). "REC-MV: REconstructing 3D Dynamic Cloth from Monocular Videos" formulated dynamic garment recovery as a joint optimization of explicit feature curves and implicit SDFs, with canonical-space template registration for open-boundary garment meshes (Qiu et al., 2023). "MOSS: Motion-based 3D Clothed Human Synthesis from Monocular Video" brought motion-conditioned Gaussian splatting into the area through Kinematic Gaussian Locating Splatting and a Surface Deformation Detector guided by matrix-Fisher modeling on $SO(3)$ (Wang et al., 2024).

Within that trajectory, MonoCloth combines explicit body–cloth decoupling, 3D Gaussian rendering, and a learned cloth-specific temporal module. A plausible implication is that it occupies an intermediate position between explicit simulation-based cloth modeling and purely appearance-driven monocular avatar methods: it does not solve cloth physics explicitly, but it also does not treat clothing as just another body surface (Jin et al., 6 Aug 2025).

2. Problem formulation and part-based decomposition

MonoCloth starts from the premise that monocular reconstruction is difficult because the input is fundamentally 2D and because loose garments, facial expressions, and articulated hands exhibit different reconstruction and deformation characteristics. The method therefore explicitly decouples the avatar into body, face, hands, and clothing, optimizing each component with a strategy matched to its motion and reconstruction complexity (Jin et al., 6 Aug 2025).

The underlying body prior is SMPL-X. MonoCloth uses an upsampled canonical mesh $v^{cano}$ and defines an identity-dependent mesh in canonical pose by learned per-vertex displacements,

$v^{id} = v^{cano} + \Delta x.$

3D Gaussians are placed at upsampled SMPL-X vertices, with attributes $\{x, c, s\}$ for position, color, and scale; opacity is fixed to 1 and the Gaussians are isotropic. The full Gaussian set is semantically decomposed as

$$G = G_{face} \cup G_{hands} \cup G_{cloth} \cup G_{body}.$$

The face and hands are anchored by strong parametric priors. $G_{face}$ is aligned to FLAME via vertex correspondences, while $G_{hands}$ is aligned to MANO/SMPL-X correspondences. Clothing is extracted through 2D clothing segmentation followed by a connectivity-based refinement, so that the cloth subset is not merely inferred as residual geometry but treated as an independently modeled component (Jin et al., 6 Aug 2025).

This decomposition is central to the method’s definition. The paper argues that a one-size-fits-all deformation model tends to produce flicker, drift, or implausible cloth behavior during animation because rigid, bone-driven body deformation differs qualitatively from soft, history-dependent cloth dynamics. MonoCloth’s architecture is therefore organized around the premise that clothing must be modeled separately if monocular avatars are to remain animatable and visually stable (Jin et al., 6 Aug 2025).

3. Canonical representation, triplane features, and dynamic deformation

MonoCloth uses a two-stage learning scheme. In Stage 1, a shared network is pretrained across multiple monocular subjects using a compact identity latent code $z_i \in \mathbb{R}^{64}$. In Stage 2, the model is fine-tuned per subject on the target monocular sequence. Canonical features are produced by a shared decoder

$T_i = D_\phi(z_i) = \{T^x_i, T^y_i, T^z_i\},$

where each triplane has shape $\mathbb{R}^{C \times H \times W}$ with $v^{cano}$0 and $v^{cano}$1 (Jin et al., 6 Aug 2025).

Per-vertex canonical features $v^{cano}$2 are obtained by bilinear sampling from the triplanes at the vertices of $v^{cano}$3. MLP decoders then map these features to per-vertex Gaussian attributes: spatial displacement $v^{cano}$4, base color $v^{cano}$5, and scale $v^{cano}$6. The canonical avatar is thus a deformable Gaussian field anchored to a canonicalized SMPL-X mesh rather than an unstructured point cloud (Jin et al., 6 Aug 2025).

Dynamic modeling is expressed in canonical space and then transferred to posed frames through skinning:

$v^{cano}$7

The paper emphasizes that offsets are predicted in canonical space and then skinned, which decouples offset learning from articulation and improves generalization across identities. Clothing is the main beneficiary of this design because it undergoes substantial non-rigid deformation, while exposed body parts such as forearms and calves are handled by a simpler pose-dependent module that predicts only small offsets (Jin et al., 6 Aug 2025).

The temporal conditioning window is

$v^{cano}$8

with $v^{cano}$9 and low sampling at 5 FPS. This explicit use of neighboring poses is intended to inject motion history into cloth deformation prediction rather than treating each frame as a purely static inverse problem (Jin et al., 6 Aug 2025).

4. CloSim, geometric supervision, and objective function

CloSim is the distinctive module of MonoCloth. The paper explicitly states that CloSim is a learned spatio-temporal simulator—not a physics solver. Its spatial component is a GCN over clothing Gaussians, and its temporal component is a GRU that encodes motion history (Jin et al., 6 Aug 2025).

For frame $v^{id} = v^{cano} + \Delta x.$0, encoded clothing features are constructed as

$v^{id} = v^{cano} + \Delta x.$1

where $v^{id} = v^{cano} + \Delta x.$2, $v^{id} = v^{cano} + \Delta x.$3, $v^{id} = v^{cano} + \Delta x.$4 is the number of clothing Gaussians, and $v^{id} = v^{cano} + \Delta x.$5. Sequence modeling then predicts residuals for position, color, and scale:

$v^{id} = v^{cano} + \Delta x.$6

During training, given $v^{id} = v^{cano} + \Delta x.$7, CloSim outputs $v^{id} = v^{cano} + \Delta x.$8, and a random supervision time $v^{id} = v^{cano} + \Delta x.$9 is chosen so that the target offset is linearly interpolated,

$\{x, c, s\}$0

with analogous interpolation for $\{x, c, s\}$1 and $\{x, c, s\}$2 (Jin et al., 6 Aug 2025).

MonoCloth uses a multi-target supervision regime. Rendered normals $\{x, c, s\}$3 and depth $\{x, c, s\}$4 are supervised against Sapiens-derived pseudo ground truth $\{x, c, s\}$5 and $\{x, c, s\}$6. The geometry loss is

$\{x, c, s\}$7

with

$\{x, c, s\}$8

and $\{x, c, s\}$9, $$G = G_{face} \cup G_{hands} \cup G_{cloth} \cup G_{body}.$$0, $$G = G_{face} \cup G_{hands} \cup G_{cloth} \cup G_{body}.$$1. Temporal smoothness is enforced through

$$G = G_{face} \cup G_{hands} \cup G_{cloth} \cup G_{body}.$$2

with $$G = G_{face} \cup G_{hands} \cup G_{cloth} \cup G_{body}.$$3 (Jin et al., 6 Aug 2025).

RGB reconstruction combines pixel, structural, perceptual, and clothing-focused terms:

$$G = G_{face} \cup G_{hands} \cup G_{cloth} \cup G_{body}.$$4

Face and hands are regularized by correspondence losses,

$$G = G_{face} \cup G_{hands} \cup G_{cloth} \cup G_{body}.$$5

The full training objective is

$$G = G_{face} \cup G_{hands} \cup G_{cloth} \cup G_{body}.$$6

Optimization uses Adam with initial learning rate $$G = G_{face} \cup G_{hands} \cup G_{cloth} \cup G_{body}.$$7. The reported practical profile is approximately 5 minutes for 3D clothing segmentation and approximately 3.5 hours for per-subject reconstruction on an RTX 4090. The paper further notes that inference is fast once trained, with real-time rendering possible via 3DGS tooling (Jin et al., 6 Aug 2025).

5. Quantitative performance, animation, and clothing transfer

MonoCloth is evaluated on NeuMan and X-Humans. On NeuMan, using masked background and metrics over Seattle/Bike/Citron/Jogging, the full method reports PSNR 33.53, SSIM 0.986, and LPIPS $$G = G_{face} \cup G_{hands} \cup G_{cloth} \cup G_{body}.$$8. On X-Humans, it reports PSNR 30.68, SSIM 0.976, and LPIPS $$G = G_{face} \cup G_{hands} \cup G_{cloth} \cup G_{body}.$$9 (Jin et al., 6 Aug 2025).

Setting	Metrics	Comparison note
NeuMan, Ours (full)	PSNR 33.53, SSIM 0.986, LPIPS $G_{face}$0	Higher than ExAvatar and 3DGS-Avatar
NeuMan, Ours (w/o pretrain)	PSNR 33.18, SSIM 0.985, LPIPS $G_{face}$1	Lower than full MonoCloth
NeuMan, ExAvatar	PSNR 31.70, SSIM 0.982, LPIPS $G_{face}$2	Baseline
NeuMan, 3DGS-Avatar	PSNR 29.75, SSIM 0.975, LPIPS $G_{face}$3	Baseline
NeuMan, GaussianAvatar	PSNR 28.90, SSIM 0.974, LPIPS $G_{face}$4	Baseline
NeuMan, Vid2Avatar-Pro	PSNR 32.71, SSIM 0.983, LPIPS $G_{face}$5	Multi-view pretrain baseline
X-Humans, Ours	PSNR 30.68, SSIM 0.976, LPIPS $G_{face}$6	Higher PSNR/SSIM than ExAvatar
X-Humans, ExAvatar	PSNR 29.41, SSIM 0.973, LPIPS $G_{face}$7	Baseline

The ablations isolate the contribution of the cloth-specific dynamic model and of geometric pseudo-supervision. On NeuMan, removing CloSim offsets and retaining only canonical LBS reduces performance to PSNR 31.93, SSIM 0.983, LPIPS $G_{face}$8. Removing temporal sampling gives PSNR 33.00, SSIM 0.985, LPIPS $G_{face}$9. Removing $G_{hands}$0 gives PSNR 32.55, SSIM 0.985, LPIPS $G_{hands}$1 (Jin et al., 6 Aug 2025). These numbers support the paper’s claim that temporal modeling is especially beneficial on continuous sequences and that animations exhibit higher stability and robustness under drastic motions.

Animation is defined by driving the avatar with a new motion sequence $G_{hands}$2 and applying

$G_{hands}$3

while rendering Gaussians with updated attributes $G_{hands}$4 (Jin et al., 6 Aug 2025). Clothing transfer follows the same decoupled logic: extract $G_{hands}$5 from a source avatar, attach it to a target body through canonical attachment and skinning weights, estimate rest-shape offsets for the target identity, apply minor local attachment refinement, and then animate the transferred garment with the target’s $G_{hands}$6 (Jin et al., 6 Aug 2025). The paper describes the resulting design as making garments plug-and-play across bodies.

6. Limitations, misconceptions, and position in the design space

A central clarification is that MonoCloth is not an explicit cloth-physics method. The paper states that CloSim does not model stretch/bend energies, friction, collisions, or self-collisions. Extreme motions, thin fabrics, or highly loose garments can therefore produce interpenetrations or underdamped behavior, and MonoCloth does not include explicit physics-based collision or self-collision energies (Jin et al., 6 Aug 2025). A common misconception is to equate its cloth simulation module with a differentiable physics solver; this is incorrect in the terminology of the paper itself.

The method also depends on pseudo 3D cues. Normals and depth are taken from Sapiens, and failure modes arise under heavy occlusions or unusual lighting. Clothing mis-segmentation or label noise may cause incorrect decoupling or artifacts despite the connectivity-based refinement. Very fast motion or severe occlusions remain difficult even with temporal smoothing and GRU memory (Jin et al., 6 Aug 2025).

These limitations distinguish MonoCloth from earlier and parallel strands of monocular cloth research. The 2020 physics-aware method embeds a simulation layer into training and explicitly handles cloth–body collisions, reducing cloth–body intersections and improving physical plausibility under weak 2D multi-view supervision (Li et al., 2020). SCARF separates body and clothing through mesh-integrated volume rendering and segmentation-driven losses, but represents clothing as a canonical-space NeRF rather than as cloth-decoupled Gaussians (Feng et al., 2022). REC-MV prioritizes explicit feature curves, multiple garment-type SDFs, and template registration for open-boundary garment meshes (Qiu et al., 2023). xCloth targets template-free textured 3D garment digitization from a single image through layered peelmaps and automatic UV atlas generation (Srivastava et al., 2022). MOSS, by contrast, uses motion-aware Gaussian control through matrix-Fisher-guided orientation and density, plus a surface deformation detector, but does not organize the avatar around MonoCloth’s body/face/hands/clothing decomposition and CloSim formulation (Wang et al., 2024).

A plausible implication is that MonoCloth occupies a specific operating point in monocular cloth capture: it favors part-aware controllability, learned temporal clothing dynamics, and efficient 3D Gaussian rendering over explicit physical energies or category-specific garment templates. That choice gives it strong animation behavior, face and hand detail, and clothing transfer capability, while leaving unresolved the hardest cases of collision-rich, topology-changing, or strongly inertial cloth motion (Jin et al., 6 Aug 2025). The paper does not state public release; its reproducibility is therefore tied to access to NeuMan and X-Humans and to reimplementation of 3D Gaussian splatting with triplane features, SMPL-X fitting, Sapiens supervision, and the CloSim GCN+GRU stack (Jin et al., 6 Aug 2025).