GarmentNet: Deep Learning for Garment Analysis

• GarmentNet is a family of deep learning architectures for garment recognition, manipulation, and simulation using RGB-D and 3D data.
• It leverages hybrid models like CNNs, Siamese/triplet architectures, and physically-inspired loss functions to ensure robust, real-time robotic interaction.
• Key applications include shape prediction, 3D draping, pose-guided synthesis, and canonical space shape completion to enhance garment analysis.

GarmentNet refers to a family of deep learning architectures and methodologies for garment recognition, manipulation, pose estimation, and physical simulation in robotic and computer vision settings. Developed across multiple research works, the term encompasses approaches for classifying, localizing, segmenting, synthesizing, and draping garments using RGB, RGB-D, or 3D point cloud data. The core GarmentNet frameworks are notable for their hybridization of geometric deep learning, convolutional neural networks, Siamese/triplet architectures, and physically-inspired loss functions, often tailored for real-time performance and robotic interaction.

1. Architectures and Variants

1.1 Continuous Robot Vision for Shape/Weight Prediction (GarNet)

GarNet, as presented in "GarNet: A Continuous Robot Vision Approach for Predicting Shapes and Visually Perceived Weights of Garments," employs a Siamese (triplet) architecture to map RGB-D video frames of garments manipulated by a robot into a low-dimensional "Garment Similarity Map" (GSM) (Duan et al., 2021). The backbone consists of a ResNet-18 feature extractor, followed by a fully connected embedding head of dimensions 512→128→32→2. Each frame is mapped to a point in $\mathbb{R}^2$; triplet loss encourages geometric and physical similarity encoding within this space. No hand-crafted features are used—the network learns to extract meaningful garment dynamics, wrinkle and silhouette information directly from the temporal stream.

1.2 Static 3D Cloth Draping (GarNet / GarNet++)

For 3D garment-body simulation, GarNet is a two-stream architecture comprising a garment stream (mesh input with point-wise MLPs, patch-wise mesh convolutions, and global pooling) and a body stream (PointNet-style processing of an unordered 3D body point cloud) (Gundogdu et al., 2018). Features from both streams are concatenated and fused to predict per-vertex translation vectors, yielding a draped garment mesh. GarNet++ augments this with curvature-preserving loss functions for enhanced high-frequency detail, including mean curvature normal and Rayleigh quotient-based local covariance constraints (Gundogdu et al., 2020).

1.3 Coarse-to-Fine Pose Synthesis and Parsing (GarmentNet in Image Synthesis)

Another GarmentNet formulation appears in pose-guided person image synthesis, where the system predicts per-pixel part parsing and segmentation via multi-scale feature-domain alignment, using learned pose flow fields between source and target pose representations (Zheng et al., 2019). This variant includes parallel encoder streams for source garment+pose and target pose, feature warping with multi-scale flows, gated fusion, and a U-Net style decoder for image or segmentation synthesis. Training objectives center on unsupervised photometric, style, and smoothness losses for flow, and per-pixel cross-entropy for parsing.

1.4 Category-Level 3D Pose Completion (GarmentNets)

GarmentNets introduces a canonical space shape completion strategy: partial RGB-D input of a grasped garment is mapped to a shared canonical pose (using NOCS representations), and a shape-completion MLP predicts a winding-number field for volumetric reconstruction in the canonical space (Chi et al., 2021). A learned warp field regresses per-vertex offsets from canonical to observation space, supporting robust correspondence-based pose estimation even from heavily occluded observations.

2. Learning Objectives and Loss Functions

2.1 Triplet Embedding Loss

In continuous robot vision, GarNet trains using classic triplet loss: $$\mathcal{L}_{\rm triplet} = \max\{0, \| h(I_p) - h(I_a) \| - \| h(I_n) - h(I_a) \| + \alpha \}$$ where $I_a$ (anchor) and $I_p$ (positive) are frames of the same garment class, while $I_n$ (negative) is from a different class. The embedding function $h(\cdot)$ projects images to $\mathbb{R}^2$ (Duan et al., 2021).

2.2 Physically Inspired Draping Losses

Cloth simulation variants minimize a composite loss, including:

• Vertex-wise fit: $$\mathcal{L}_{\rm vertex} = \frac{1}{N} \sum_i \| G^G_i - G^P_i \|^2$$
• Interpenetration: penalizes garment points inside body based on local normal dot products
• Normal alignment: penalizes angular deviation between predicted and GT facet normals
• Bending: preserves edge-lengths among second-order mesh neighbors
• Curvature losses (GarNet++): mean-curvature normal and multi-scale Rayleigh quotient terms for local geometric fidelity (Gundogdu et al., 2018, Gundogdu et al., 2020).

2.3 Multi-Task and Cross-Entropy Losses

Segmentation-based GarmentNets combine per-pixel cross-entropy for parsing, GAN/patch-based adversarial loss for photo-realism, perceptual (VGG) and style losses, and photometric/smoothness for unsupervised flow estimation (Zheng et al., 2019).

2.4 Canonical Space Supervision

Category-level shape completion employs classification loss for per-point canonical coordinate regression and MSE for winding number prediction and warp fields (Chi et al., 2021).

3. Datasets, Input Modalities, and Preprocessing

GarmentNet systems typically exploit a combination of synthetic and real datasets:

• 20-instance RGB-D video datasets for robotic manipulation (5 shape classes; crumpled vs. hanging) (Duan et al., 2021).
• EPFL GarmentSim and Wang et al. T-shirt datasets for 3D draping (meshes, cutting parameters) (Gundogdu et al., 2018).
• CloPeMa Garment dataset (3,330 images, 9 classes, 27 landmark types) for landmark localization and folding (Gomes et al., 2019).
• CLOTH3D category templates for canonicalization (Chi et al., 2021).

Input preprocessing ranges from segmentation of depth maps, mesh skinning (dual quaternion), RGB resizing, color jitter, to learned per-pixel part representations; augmentation varies by task and is often omitted in real-time robotic vision settings.

4. Evaluation Metrics and Experimental Results

4.1 Continuous Vision Classification

GarNet reaches 92.0% shape classification accuracy and 95.5% predicted weight classification in robotic video compared to prior bests of 70.8% and 48-67% (see table). Decision-point (temporal mean in the GSM) outperforms per-frame prediction by 14–16% (Duan et al., 2021).

4.2 3D Draping

On 3D garment-body tasks, GarNet achieves sub-centimeter mean vertex errors (0.88–0.97 cm) and normal errors of 5.6–9.2° across multiple garment types, providing ∼100× speedup over physics-based simulation while preserving wrinkle and fold realism. GarNet++ further reduces curvature discrepancy and prevents body interpenetration (Gundogdu et al., 2018, Gundogdu et al., 2020).

4.3 Parsing and Landmark Detection

In the CloPeMa dataset, GarmNet-B yields a 17.8% classification+localization error (down from 56.7% baseline), with landmark mAP of 36.2%. Bridge fusion of local landmark scores into the global branch increases robustness and stability (Gomes et al., 2019).

4.4 Category-Level Pose

GarmentNets reports average Chamfer distance $D_c$ of 1.70 cm and correspondence distance $D_n$ of 5.73 cm across six garment categories, outperforming nearest-neighbor retrieval and direct task-space approach by substantial margins (Chi et al., 2021).

5. Key Mechanisms and Early-Stop Decision Rules

5.1 Manifold Embedding and Early Exit

GarNet’s GSM-based approach fits clusters (per known class) using kernel density estimation and monitors the rolling mean embedding (Decision Point). When $\geq$80% of streaming observations lie within a class confidence region, classification stops (early decision), reducing latency without requiring the garment to reach an unoccluded state (Duan et al., 2021).

5.2 Feature Fusion and Cross-Scale Alignment

Hybrid architectures exploit early-, mid-, and late-fusion: body and garment features, or local and global cues, are concatenated, max-pooled, or cross-attended at each processing block. This design supports synergy between fine-grained structure (sleeves, seams) and holistic pose or class context (Gundogdu et al., 2018, Gomes et al., 2019, Zheng et al., 2019).

5.3 Multi-Scale Flow for Pose Transfer

Pose-guided GarmentNet propagates information via learned multi-scale flow fields, employing gated attention at all decoder levels to reconcile ambiguities and support consistent fine-to-coarse feature synthesis (Zheng et al., 2019).

6. Limitations and Future Directions

Principal limitations include:

• Fixed, closed-set category handling for classification tasks; open-set or one-shot adaptation remains unsolved (Duan et al., 2021).
• Requirement for large, labeled mesh datasets with dense ground truth (especially for canonical completion) (Chi et al., 2021).
• Curvature losses in GarNet++ remain numerically sensitive and do not guarantee temporal coherence in dynamic manipulations (Gundogdu et al., 2020).
• Real-world deployment scalability is affected by background complexity, occlusion, and intra-class variance (e.g., robotic folding in cluttered scenes) (Gomes et al., 2019).

Future research involves online category adaptation, domain transfer (simulation-to-real), dynamic sequence modeling, and integrating physically differentiable simulation within end-to-end learning. Enhanced manipulation policies, plug-and-play garment models, and fully unsupervised canonical space learning without mesh or winding-number supervision represent open directions.

---

References:

• GarNet: A Continuous Robot Vision Approach for Predicting Shapes and Visually Perceived Weights of Garments (Duan et al., 2021)
• GarNet: A Two-Stream Network for Fast and Accurate 3D Cloth Draping (Gundogdu et al., 2018)
• GarNet++: Improving Fast and Accurate Static3D Cloth Draping by Curvature Loss (Gundogdu et al., 2020)
• GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion (Chi et al., 2021)
• Unsupervised Pose Flow Learning for Pose Guided Synthesis (Zheng et al., 2019)
• GarmNet: Improving Global with Local Perception for Robotic Laundry Folding (Gomes et al., 2019)