---
title: 'ReMu: Layered 3D Clothed Human Reconstruction'
url: https://www.emergentmind.com/topics/remu
type: topic
---

# ReMu: Layered 3D Clothed Human Reconstruction

ReMu is a training-free reconstruction pipeline for recovering a multi-layer 3D clothed human from **Image Layers**: a set of monocular RGB photographs of the same subject captured while progressively wearing different subsets of garments. Unlike single-surface clothed-human reconstruction, ReMu reconstructs the body, inner garments, and outer garments as separate layers, aligns them in a shared canonical body coordinate system, applies collision-aware penetration removal, and then refines surfaces with neural unsigned distance fields, yielding nearly penetration-free layered geometry for downstream tasks such as clothing simulation and avatar creation [2508.01381].

## 1. Problem setting and capture model

ReMu addresses a reconstruction setting that prior clothed-human methods generally do not solve explicitly: recovering a **multi-layer** dressed human in which the body, inner garments, and outer garments are represented as distinct 3D layers rather than being fused into one surface. The input modality is a new capture setup called **Image Layers**, where a subject is photographed by a **single static RGB camera** multiple times while progressively adding clothing layers. One image may show only a lower garment, another the lower plus an inner garment, and another the full stack including an outer garment. The camera need not be calibrated, and the method does not use the camera position or matrix [2508.01381].

This setting is designed to expose geometry that is otherwise hidden in a single image. Inner garments that become occluded in the final dressed state remain visible in earlier captures, which reduces the ambiguity that normally forces strong garment-category priors. At the same time, Image Layers introduces a coupled geometric problem: poses across captures are only approximately matched; independently reconstructed garments do not align naturally; and naive stacking of reconstructions produces collisions, misalignment, and broken boundaries. ReMu is formulated around these three difficulties.

The method is explicitly **template-free** and **category-agnostic**. It does not assume predefined garment templates, category-specific latent spaces, sewing-pattern priors, or synthetic-data priors tied to known classes. This is the basis for its reported behavior on clothing categories such as **long coats and dresses**, where category-specific methods are described as struggling while ReMu reconstructs the garments without retraining or category adaptation [2508.01381].

## 2. Reconstruction pipeline and garment isolation

For each image layer \(I_k\), ReMu first estimates a body model \(B_k\) with **SMPLer-X**, then refines the body pose by minimizing 2D keypoint reprojection error. Given this body estimate, it uses **SiTH** as an off-the-shelf single-view clothed human reconstruction model to infer a watertight textured clothed mesh \(M_k\). SiTH fits SMPL-X, hallucinates a back view with a diffusion model, and reconstructs a watertight clothed-human mesh from front/back image features and body guidance. At this stage, however, \(M_k\) is still a fused clothed-human surface rather than an isolated garment layer [2508.01381].

Garment separation is performed in 3D by combining mesh rendering, 2D segmentation, and back-projection. ReMu renders synthetic views of \(M_k\), runs **GroundedSAM** on those views to segment the target clothing in 2D, and projects the labels back onto the mesh. In the implementation, the method renders **60 views**, sampled circularly with **\(10^\circ\)** horizontal spacing and **\(30^\circ\)** spacing for upper and lower views. Vertex labels are assigned by majority vote over projected masks; disconnected fragments are filtered; and labels are smoothed to obtain a 3D garment region \(S_k\) on \(M_k\). This step converts a generic clothed-body reconstruction into an explicit layer-specific garment region [2508.01381].

The overall pipeline then follows a fixed sequence: reconstruct a clothed mesh and parametric body for each image layer, isolate the garment of interest for that layer, warp all garments into a common canonical coordinate system, perform collision-aware penetration removal from inner to outer layers, and finally fit **neural unsigned distance fields (UDFs)** to refine boundaries and extract the final meshes. In the authors’ formulation, the central contribution is not a new monolithic network but a hybrid pipeline that combines explicit meshes with implicit neural fields.

## 3. Canonical-body alignment and layered collision handling

After garment extraction, the reconstructed meshes are still mutually inconsistent because the subject may have slightly different poses across captures. ReMu resolves this by defining a shared **canonical body pose** \(B_c\), typically a T-pose, and using it as the common coordinate system for all layers. The body is represented by **SMPL-X**, while each garment initially remains a segmented region on a watertight reconstructed mesh. ReMu brings each mesh into canonical pose using **inverse linear blend skinning (inverse LBS)** [2508.01381].

The forward LBS relation for a vertex \(\mathbf{v}_k\) in observed pose and its canonical counterpart \(\mathbf{v}_k^c\) is
\[
\mathbf{v}_k = LBS (\mathbf{v}_k^c) = \sum_{i=1}^{21} w_i^c \mathbf{B}_{ki} \mathbf{v}_k^c,
\]
where \(\mathbf{B}_{ki}\) is the bone transformation matrix for joint \(i\) under pose \(B_k\), and \(w_i^c\) are skinning weights. The inverse mapping is
\[
\mathbf{v}_k^c = LBS^{-1} (\mathbf{v}_k) = \left(\sum_{i=1}^{21} w_i \mathbf{B}_{ki}\right)^{-1} \mathbf{v}_k.
\]
The skinning weights are obtained by querying the nearest vertex on the SMPL-X body. This canonicalization is the mechanism that makes consistent geometric reasoning about “inner” and “outer” garments possible [2508.01381].

ReMu then performs its central **collision-aware optimization**, implemented as an explicit geometric procedure rather than as a learned differentiable loss. The canonical body \(B_c\) is treated as layer \(0\), with \(M'_0 = B_c\), and garments are processed sequentially from inner to outer. For each garment \(k \ge 1\), the goal is to obtain a corrected mesh \(M'_k\) that lies outside \(M'_{k-1}\) while preserving the original garment shape whenever possible.

For each vertex \(\mathbf{v}_j^k\), the method finds the nearest point \(\mathbf{b}_j^k\) on the body surface and uses the body normal \(\mathbf{n}_j^k\) there as the only allowed displacement direction:
\[
\ell_j^k = \mathbf{b}_j^k + t \mathbf{n}_j^k.
\]
If \(\mathbf{v}_j^k \notin S_k\), the vertex is moved onto the surface of the previous layer \(M'_{k-1}\). If \(\mathbf{v}_j^k \in S_k\) and is not inside \(M'_{k-1}\), it is left unchanged so that loose or draped geometry is preserved. If it lies inside \(M'_{k-1}\), the method computes the intersection parameter \(t^*\) of \(\ell_j^k\) with \(M'_{k-1}\) and moves the vertex just outside the previous layer by a small offset \(\varepsilon\):
\[
\mathbf{v}^{k'}_j =
\begin{cases}
\mathbf{b}^k_j + (t^* + \varepsilon) \mathbf{n}^k_j, & \text{if } \mathbf{v}^{k'}_j \in S_k \text{ and inside of } M'_{k-1}, \\
\mathbf{b}^k_j + t^* \mathbf{n}^k_j, & \text{otherwise}.
\end{cases}
\]
In implementation, the garment thickness parameter for this penetration-removal stage is fixed to **2 mm**, and mesh queries are handled with **NVIDIA Kaolin**. The result is an explicitly nested layered geometry in which body, inner garments, and outer garments are ordered by construction rather than by soft penalty terms [2508.01381].

## 4. Implicit UDF refinement and mesh extraction

The paper emphasizes two artifacts introduced by the explicit mesh stages. First, boundaries obtained by projecting 2D masks back to 3D can be noisy or fragmented. Second, penetration-removal shifts are discontinuous and can damage local smoothness. ReMu addresses both issues by re-representing each corrected garment as a **neural implicit unsigned distance field** and remeshing it afterward [2508.01381].

For garment \(k\), the UDF is defined as
\[
f_k : \mathbb{R}^{3} \rightarrow \mathbb{R}_{\geq0}, \qquad f_k (\mathbf{x}) = d_k.
\]
In the main paper, the fitting objective is written
\[
\mathcal{L}_{udf} = \frac{1}{|N_l|} \sum_{k=1}^{N_l} \mathbb{E}_{\mathbf{x}_k\sim S_k}\Bigl[ \left \| f_k(\mathbf{x}_k) - d_k \right \|_{2}^{2}\Bigr].
\]
The supplementary gives the training formulation actually used in implementation. For each corrected mesh \(M'_k\) and garment region \(S_k\), points are sampled on the surface, near the surface by Gaussian perturbation, and uniformly in a bounding box. Points whose nearest surface point is not on \(S_k\) are discarded; the ground-truth unsigned distance \(d_k^{gt}\) to \(M'_k\) is then computed; and training uses a clamped loss,
\[
\mathcal{L}_{UDF} = \frac{1}{K} \sum_{k=1}^{K} \mathbb{E}_{x_k \sim S_k} \left[ \left\| \min(f_k(x_k), \delta) - \min(d_{k}^{gt}, \delta) \right\|_2^2 \right].
\]
The field is unsigned rather than signed because garments are open or thin surfaces, and unsigned distance is described as more robust for such geometry [2508.01381].

The network for each garment UDF is a **4-layer MLP** with hidden dimensions **128, 256, 256, 128** and **ReLU** activations, trained with **Adam** at learning rate \(10^{-4}\). The paper also studies positional encodings inspired by NeRF,
\[
PE(\mathbf{x}) =
\begin{cases}
\sin(2^i\mathbf{x}) \\
\cos(2^i\mathbf{x})
\end{cases},
\qquad i = 0, \ldots, N - 1,
\]
and chooses **\(N=4\)** as a compromise between preserving wrinkles and avoiding noise reconstruction. After fitting, each garment is extracted with **Marching Cubes**. The supplementary states that the extraction threshold is **3 mm**, producing a **double-sided mesh with small thickness**, and also describes a rendering-oriented postprocess that flips back-face normals for faces whose vertices lie sufficiently behind the body along the viewing direction [2508.01381].

## 5. Evaluation, metrics, and ablation behavior

The evaluation uses the **4D-DRESS** dataset, which contains multi-view images and 3D scans with semantic clothing labels for 32 clothed subjects. The authors randomly select 12 subjects to build a multilayer reconstruction benchmark, extract ground-truth lower, inner, and outer garment meshes from semantic labels, and deform lower and inner garments to the outer-garment pose for evaluation. For fairness, all compared methods are given **ground-truth poses and labels**, so the benchmark isolates clothing reconstruction rather than body fitting. ReMu is compared against **SMPLicit**, **ClothWild**, and **ISP**, together with adapted “+” variants that are allowed to use the same Image Layer inputs [2508.01381].

The reported metrics are **Chamfer Distance (CD)** in millimeters, **Normal Consistency (NC)**, and **Intersection Ratio (IR)**. IR is introduced as a visibility-based measure of interpenetration: garments are rasterized from front and back views, \(A\) denotes the visible area of the garment alone, and \(\hat{A}\) denotes the visible area after stacking all inner garments and body. Higher IR indicates that more of the garment is hidden by intersections with inner layers.

Quantitatively, ReMu achieves **lower garments**: CD \(= 8.422\) mm, NC \(= 0.927\), IR \(= 0.052\%\); **inner garments**: CD \(= 14.14\) mm, NC \(= 0.855\), IR \(= 0.204\%\); and **outer garments**: CD \(= 18.53\) mm, NC \(= 0.807\), IR \(= 0.485\%\). The paper stresses that these IR values remain below **0.5%** for all garment categories, whereas original single-image baselines can show much higher penetration, including **SMPLicit** with outer IR \(= 6.093\%\) and **ClothWild** with inner IR \(= 0.756\%\). It also notes that even the inverse-LBS-aligned **ground-truth 4D-DRESS meshes** are not penetration-free, with reference IR values of \(5.762\%\) for lower, \(6.776\%\) for inner, and \(9.997\%\) for outer. The paper’s interpretation is that ReMu is not merely approximating aligned scan geometry; it is producing layered geometry that is more directly usable for simulation [2508.01381].

The ablation study isolates three stages: inverse LBS only, inverse LBS plus penetration removal, and the full model including garment refinement. With **inverse LBS only**, penetrations are severe: lower IR \(= 17.06\%\), inner IR \(= 35.70\%\), outer IR \(= 27.82\%\). Adding **penetration removal** reduces them to lower \(0.002\%\), inner \(0.075\%\), and outer \(0.060\%\). Adding **UDF refinement** improves CD and NC but slightly raises IR: lower \(0.002\%\to 0.052\%\), inner \(0.075\%\to 0.204\%\), outer \(0.060\%\to 0.485\%\). The authors explicitly frame this as a tradeoff: explicit collision correction gives nearly perfect separation, while implicit refinement restores surface quality at the cost of a small reintroduction of intersections. They argue that the remaining IR is practical for Blender-based simulation [2508.01381].

## 6. Relations to prior work, scope, and nomenclature

ReMu is positioned against earlier single-image multilayer clothing methods such as **BCNet**, **SMPLicit**, **ClothWild**, and **ISP**, which are described as relying on predefined garment templates, category-specific latent spaces, synthetic training data, or sewing-pattern priors. ReMu departs from this design space by reconstructing layers from actual observations across progressive dressing states. This is why the paper repeatedly describes it as **template-free** and **category-agnostic**, and why it highlights qualitative generalization to garments such as dresses and long coats without retraining [2508.01381].

The method nonetheless has clear capture and modeling assumptions. It requires a **set of image layers** taken with a **fixed monocular RGB camera**, one image per garment layering state, and assumes the subject can be captured repeatedly while gradually adding clothing layers. In the reported implementation, the authors used an **iPhone 13 Pro Max on a tripod**, cropped images to \(1024 \times 1024\), and removed backgrounds with Segment Anything. The current formulation is also described as tailored to subjects in roughly **T/A pose**. A practical limitation is that multi-state capture may be inconvenient in some settings; another is that the UDF refinement stage does not include explicit collision constraints, which is why small penetrations can reappear after refinement. The paper points to extending the idea to single-view videos as future work [2508.01381].

A common misconception is to treat ReMu as a single learned network. The paper explicitly argues otherwise: its main contribution is a reconstruction **formulation** built from a shared canonical-body representation, explicit penetration-aware layering, and implicit UDF-based refinement. This suggests that ReMu is best understood as a hybrid geometric-neural system rather than as a category-conditioned predictor.

The name also requires disambiguation. **“ReMu”** in this context refers to **“ReMu: Reconstructing Multi-layer 3D Clothed Human from Image Layers”** [2508.01381]. It is distinct from **“ReMU: Regional Minimal Updating for Model-Based Derivative-Free Optimization”** [2504.03606] and from **ReMU, “Response Matrix Utilities,”** a Python package for response-matrix-centred forward folding in cross-section measurements [1903.06568]. These are unrelated uses of a similar string, and only the first denotes the layered clothed-human reconstruction method described here.

Source: https://www.emergentmind.com/topics/remu