---
title: 'LaGarNet: Latent Model for Garment Flattening'
url: https://www.emergentmind.com/topics/lagarnet
type: topic
---

# LaGarNet: Latent Model for Garment Flattening

LaGarNet denotes a **goal-conditioned latent world model for garment flattening** that addresses **single-gripper pick-and-place (PnP) flattening of complex garments** such as long-sleeved T-shirts, trousers, skirts, and dresses. In [2508.17070], it is formulated as an **offline model-based deep RL** system built around a **goal-conditioned recurrent state-space model (GC-RSSM)**, a **coverage-alignment reward**, and a dataset collection procedure that combines a **Diffusion Policy** with a **mask-biased random policy**. The method is presented as the **first successful application of state-space models to complex garments**, with the central motivation that garment manipulation is **partially observable**, deformable, and multi-step, while the robot must plan over long horizons from image observations.

## 1. Problem setting and scope

LaGarNet is designed for **pick-and-place garment flattening**: given an image of a crumpled garment and a goal flattened image, the robot must repeatedly pick a point on the cloth and place it elsewhere to progressively flatten the garment [2508.17070]. The task is difficult because cloth is **deformable** and **self-occluding**; the robot sees only a partial top-down RGB/D observation; action effects are long-horizon and highly nonlinear; and garments introduce sleeves, legs, openings, and topology changes that are difficult for engineered methods.

The method targets **complex garments**, not only simpler fabrics such as towels. Four garment types are used: **long-sleeved T-shirt**, **trousers**, **dress**, and **skirt**. The paper’s objective is not merely to solve a single garment instance, but to train a **single policy** that generalizes across garment categories. This is paired with an explicit attempt to reduce assumptions that were prominent in previous model-based cloth systems, including **mesh reconstruction / mesh dynamics**, **special garment-specific oracles**, **corner-biased or hand-crafted sampling**, and **reward engineering tightly coupled to a specific cloth type**.

A central design premise is that **goal-conditioned recurrent state-space models (GC-RSSMs)** are well matched to the problem. Two reasons are emphasized. First, because the robot never sees the full true cloth state, a latent recurrent model is useful for belief tracking under **partial observability**. Second, flattening is sequential, so world models support imagination-based planning under **long-horizon dynamics**. The action primitive is **quasi-static high-level** PnP rather than fine velocity control, so the representation problem is to encode the evolving cloth configuration compactly and predict the effect of discrete pick-place actions.

## 2. GC-RSSM formulation and latent dynamics

The base recurrent state-space model follows the standard world-model decomposition into deterministic and stochastic latent states [2508.17070]. The recurrent latent dynamics are written as
\[
\mathbf{h}_t = f(\mathbf{h}_{t-1}, \mathbf{z}_{t-1}, \mathbf{a}_{t-1}),
\]
with posterior and prior stochastic states
\[
\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t),
\qquad
\tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t),
\]
where \(\mathbf{x}_t\) is the observation, \(\mathbf{a}_t\) the action, \(\mathbf{h}_t\) the deterministic recurrent state, and \(\hat{\mathbf{z}}_t,\tilde{\mathbf{z}}_t\) the posterior and prior stochastic states.

LaGarNet extends this into a **goal-conditioned** RSSM by conditioning both inference and prediction on the goal image \(\mathbf{g}\):
\[
\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t, \mathbf{g}),
\]
\[
\tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{g}).
\]
The stated purpose is to prevent the recurrent model from “forgetting” the target and to make latent rollouts goal-aware.

The model factorization is given as
\[
p(\mathbf{x}_{1:T}, \mathbf{z}_{1:T} \mid \mathbf{a}_{1:T}, \mathbf{g}) =
\prod_{t=1}^T p(\mathbf{x}_t \mid \mathbf{z}_t)\, p(\mathbf{z}_t \mid \mathbf{z}_{t-1}, \mathbf{a}_{t-1}, \mathbf{g}),
\]
\[
q(\mathbf{z}_{1:T} \mid \mathbf{x}_{1:T}, \mathbf{a}_{1:T}, \mathbf{g}) =
\prod_{t=1}^T q(\mathbf{z}_t \mid \mathbf{x}_{1:t}, \mathbf{a}_{1:t}, \mathbf{g}).
\]

The latent state is intended to represent the **hidden physical configuration of the cloth**. More specifically, \(\mathbf{z}_t\) is the **stochastic latent state** encoding uncertain cloth configuration, while \(\mathbf{h}_t\) is the **deterministic recurrent memory** that propagates temporal information. Together they act as a learned belief state over garment deformation. The qualitative reconstruction analysis reported in the paper indicates that this latent state is particularly effective at tracking **sleeves and garment boundaries** over imagined future steps. This suggests that the goal-conditioning does not merely specify a terminal objective; it also shapes the intermediate latent geometry used for planning.

The goal is provided as a **goal image** \(\mathbf{g} \in \mathbb{R}^{C \times H \times W}\), encoded by the same visual encoder used for the current observation. The resulting **goal embedding** \(\mathbf{e}_g\) is injected into the latent inference and transition models. A practical component is **goal augmentation**: rather than requiring a canonical goal image at test time, the goal input is subjected to step-wise augmentation and Gaussian translation
\[
\mathcal{N}(0, 0.2^2).
\]
This is used to bridge sim-to-real appearance gaps and to avoid requiring a perfectly canonical goal state in real experiments.

## 3. Objectives, reward design, and data collection

LaGarNet combines world-model learning with reward learning [2508.17070]. The training objective is stated as maximizing the log-likelihood of observations and rewards conditioned on actions and goal:
\[
\arg\max_{p_{\mathbf{x}}, p_r} \ln p_{\mathbf{x}}(\mathbf{x}_{1:T} \mid \mathbf{a}_{1:T}, \mathbf{g}) + \ln p_r(r_{1:T} \mid \mathbf{a}_{1:T}, \mathbf{g}).
\]
Using the RSSM factorization, the loss is written as
\[
\mathcal{L} = \sum_{t=1}^T \Bigg(
- \mathbb{E}_{\mathbf{z}_t}\Big[ \ln p(\mathbf{x}_t \mid \mathbf{z}_t) + \ln p(r_t \mid \mathbf{z}_t) \Big]
+ \mathbb{E}_{\mathbf{z}_{t-1}}\Big[
KL\big[ q(\mathbf{z}_t \mid \mathbf{x}_{1:t}, \mathbf{a}_{1:t-1}, \mathbf{g})
\;||\;
p(\mathbf{z}_t \mid \mathbf{z}_{t-1}, \mathbf{a}_{t-1}, \mathbf{g}) \big]
\Big]
\Bigg).
\]
The learned components therefore include observation reconstruction, reward prediction, and KL alignment between posterior and prior latent dynamics. The implementation notes include **no gradient stop on goals during training**, **gradient stop for overshooting loss**, and an increased model capacity relative to PlaNet-ClothPick, with deterministic, stochastic, and hidden dimensions raised from \((200, 30, 200)\) to \((300, 60, 300)\), alongside **two extra linear layers in the reward head**.

The **coverage-alignment reward** is a core part of the system. It uses step-wise **coverage improvement** together with change in **Max IoU** between current and goal cloth masks. A SpeedFolding-style approximation is defined as
\[
\mathcal{R}_{SFA} = \max\left(\tanh(\alpha \times \mathcal{R}_{dc} + \beta \times \mathcal{R}_{du}), 0\right),
\]
with
\[
\alpha = 1, \qquad \beta = 2.
\]
Here, \(\mathcal{R}_{dc}\) is the delta coverage reward and \(\mathcal{R}_{du}\) is the delta Max IoU reward. The reward is then modified to preserve flattened states and discourage disturbing near-success configurations:
\[
\hat{\mathcal{R}}_{CA} =
\begin{cases}
0 & \text{if } NC - R_{dC} > 0.9 \text{ and } NC < 0.9 \\
b & \text{if } NC \geq 0.95 \\
\mathcal{R}_{SFA} & \text{otherwise}
\end{cases}
\]
with
\[
b = 0.7.
\]
The reported motivation is that this reward can be used in both simulation and real-world settings, unlike rewards that require oracle particle distances or special canonicalization.

The dataset collection strategy is treated as a separate novelty. The pipeline is: collect a small number of human demonstrations, train a **Diffusion Policy** on those demonstrations, and then mix trajectories from that learned policy with those from a **mask-biased random policy**. The mask-biased random policy samples the pick pixel **uniformly from the cloth mask** and the place pixel from the whole image, avoiding impossible grasps while preserving broad exploration. In the comparison setup, the diffusion policy is trained with **50 human demonstrations**, and the LaGarNet family is trained on about **300k transitional steps** from the proposed collection scheme. The paper further reports that the exact ratio between diffusion and random data does not strongly affect NC or Max IoU, but it does affect success rate. This suggests that exploration coverage and data quality play distinct roles in world-model training and downstream planning.

## 4. Planning pipeline, environments, and evaluation protocol

At planning time, the GC-RSSM is used inside **model predictive control (MPC)** to choose actions that maximize predicted flattening reward [2508.17070]. The complete system therefore consists of a learned latent dynamics model, a learned reward predictor, and an online planner operating over imagined rollouts.

The simulation environment is taken from **Canberk et al. / ClothFunnels**, with extensions for **misgrasping** and **multi-layer grasping**. These extensions are introduced because naive parallel grippers can fail in the real world, and the simulation is intended to reflect that. Garments are drawn from the **Cloth3D** dataset. The common simulation settings are: scale factor **0.8**, camera height **2 m**, dark background, cloth stiffness parameters **stretch = 0.75**, **bend = 0.02**, **shear = 0.02**, cloth mass **0.5 kg**, and evaluation on **30 hard initial states**.

The real-world system uses a **UR3e** single-arm setup with an **Intel RealSense D435i** camera at **720 × 1280**, mounted **0.72 m above the table**. The workspace is **ring-shaped**, with far radius \(r_f = 0.54\) and near radius \(r_n = 0.24\). To connect training and deployment, the system uses a **workspace transfer heuristic** mapping the square observation window used during training to the feasible ring-shaped robot workspace. The gripper hardware is also modified by replacing a tweezer-style extension with a **3D-printed cylinder extension** to improve gripping heavier garments.

The evaluation compares LaGarNet against **ClothFunnels**, **PlaNet-ClothPick**, **JA-TN**, **Diffusion Policy**, and human reference performance. The reported metrics are **NC** (Normalized Coverage), **NI** (Normalized Improvement), **Max IoU**, and **SR** (success rate). In simulation, success is defined by reaching **90% NC** and **80% Max IoU**. In real-world experiments, two success variants are used: \(SR^{90}_{80}\) and \(SR^{90}_{0}\).

## 5. Quantitative performance and ablations

The reported quantitative results indicate that LaGarNet is strong in both simulation and real-world settings [2508.17070]. In simulation, for long-sleeved T-shirts, it significantly outperforms ClothFunnels, JA-TN, Diffusion Policy, and PlaNet-ClothPick. Representative all-garment results at **20 steps** are **NC ≈ 89.8%**, **NI ≈ 79.5%**, **Max IoU ≈ 74.7%**, and **SR = 18/30**. At **30 steps**, the corresponding figures are **NC ≈ 94.2%**, **NI ≈ 87.2%**, **Max IoU ≈ 79.5%**, and **SR = 23/30**. The paper further states that the single-policy LaGarNet obtains **above 90% NC across all garment types** in simulation.

The same results also delineate the model’s residual weaknesses. In simulation, LaGarNet still struggles to exceed **85% Max IoU**, and the paper notes that it appears to overfit somewhat to **skirts** and **dresses**, with weaker performance on **T-shirts** and **trousers** in the all-garment setting. This indicates that strong coverage does not automatically imply fine geometric alignment.

In the real world, LaGarNet is reported to substantially improve over PlaNet-ClothPick. At **10 steps**, the total performance is **NC ≈ 83.3%**, **NI ≈ 70.3%**, **Max IoU ≈ 75.8%**, and **\(SR^{90}_{80} = 16/40\)**. At **20 steps**, it reaches **NC ≈ 87.7%**, **NI ≈ 77.6%**, **Max IoU ≈ 79.2%**, **\(SR^{90}_{80} = 17/40\)**, and **\(SR^{90}_{0} = 20/40\)**. By garment type, **skirts** are described as the easiest and often best-performing category, whereas **dresses** are the hardest. Human policies still outperform LaGarNet overall, but the paper states that LaGarNet closes a large fraction of the gap.

The ablations are organized around three claims. First, the **GC-RSSM matters**: the goal-conditioned recurrent model yields better prior reconstructions and better planning than non-goal-conditioned variants. Second, the **coverage-alignment reward matters**: reward variants from ClothFunnels, Learning2Unfold, or SpeedFolding approximations are weaker. Third, **data collection matters**: the mask-biased random policy is crucial for success, and the diffusion policy further improves performance when more data is available. Qualitative results align with these ablations: predicted rewards increase over planning iterations, action distributions become more refined, and the GC-RSSM better tracks sleeve structure than other RSSM variants.

The paper’s broader empirical claim is that LaGarNet reaches **state-of-the-art levels for single-gripper garment flattening** and compares favorably to earlier mesh-based methods such as **VCD** and **MEDOR**. A plausible implication is that, for this task class, carefully trained latent world models can approach the performance of more heavily engineered mesh-centric systems without inheriting all of their structural assumptions.

## 6. Relation to prior approaches, limitations, and naming ambiguity

LaGarNet’s novelty claim is specific: it is **not** presented as the first world model for cloth in general, but as the first state-space model that works on **complex garments** rather than simpler fabrics [2508.17070]. Relative to prior mesh-based systems, the paper claims reduced inductive bias in several forms: **no explicit 3D mesh reconstruction at test time**, **no garment-specific expert oracle policies**, **no corner-biased sampling**, **no dependence on oracle particle distances for reward**, **no explicit handcrafted mesh dynamics**, and **less reliance on strong canonical shape assumptions**. By contrast, methods such as VCD and MEDOR are characterized as relying on **mesh reconstruction**, **explicit cloth topology priors**, and **garment-specific reasoning**.

Several limitations are stated explicitly. LaGarNet struggles with garments that are **too soft** or exhibit **high internal friction**; it has difficulty with **wide-open trousers/skirt structures** and seam-line reasoning; it still struggles to select and untwist **slim parts** such as sleeves and legs; it can disrupt already nearly flattened cloth; it does not distinguish front from back of garments; and all-garment training can underfit some garment types while overfitting others. Real-world performance remains below human performance. These limitations constrain any interpretation of the method as a complete solution to garment manipulation.

The name itself also requires disambiguation. In [2508.17070], **LaGarNet** refers to the garment-flattening GC-RSSM system described above. A similarly spelled but distinct term is **LaguerreNet**, a spectral GNN filter based on continuous generalized Laguerre polynomials for heterophily and over-smoothing in graph neural networks [2511.15328]. Another unrelated term is **LA-SDG**, the **learn-and-adapt stochastic dual gradient** method for online network resource allocation, which is also described in the supplied material as being called “LaGarNet” [1703.01673]. In current usage within the provided sources, however, the standalone title **“LaGarNet”** designates the garment-flattening model of [2508.17070].

Taken together, these properties position LaGarNet as a latent-dynamics, goal-conditioned, model-based framework for **single-arm PnP garment flattening**. Its defining contribution is the combination of **goal-conditioned latent dynamics**, a **coverage-alignment reward**, a **general data-collection strategy**, and a **workspace transfer heuristic**, with evidence that this combination is sufficient to make state-space models viable for complex garment manipulation.

Source: https://www.emergentmind.com/topics/lagarnet