LaGarNet: Latent Model for Garment Flattening

• LaGarNet is a goal-conditioned latent world model designed to flatten complex garments using single-gripper pick-and-place actions.
• It employs a GC-RSSM framework combined with a coverage-alignment reward and a hybrid data collection strategy to enhance planning and control.
• The system leverages offline deep RL and MPC-based planning to manage partial observability and nonlinear deformable cloth dynamics.

LaGarNet denotes a goal-conditioned latent world model for garment flattening that addresses single-gripper pick-and-place (PnP) flattening of complex garments such as long-sleeved T-shirts, trousers, skirts, and dresses. In (Kadi et al., 23 Aug 2025), it is formulated as an offline model-based deep RL system built around a goal-conditioned recurrent state-space model (GC-RSSM), a coverage-alignment reward, and a dataset collection procedure that combines a Diffusion Policy with a mask-biased random policy. The method is presented as the first successful application of state-space models to complex garments, with the central motivation that garment manipulation is partially observable, deformable, and multi-step, while the robot must plan over long horizons from image observations.

1. Problem setting and scope

LaGarNet is designed for pick-and-place garment flattening: given an image of a crumpled garment and a goal flattened image, the robot must repeatedly pick a point on the cloth and place it elsewhere to progressively flatten the garment (Kadi et al., 23 Aug 2025). The task is difficult because cloth is deformable and self-occluding; the robot sees only a partial top-down RGB/D observation; action effects are long-horizon and highly nonlinear; and garments introduce sleeves, legs, openings, and topology changes that are difficult for engineered methods.

The method targets complex garments, not only simpler fabrics such as towels. Four garment types are used: long-sleeved T-shirt, trousers, dress, and skirt. The paper’s objective is not merely to solve a single garment instance, but to train a single policy that generalizes across garment categories. This is paired with an explicit attempt to reduce assumptions that were prominent in previous model-based cloth systems, including mesh reconstruction / mesh dynamics, special garment-specific oracles, corner-biased or hand-crafted sampling, and reward engineering tightly coupled to a specific cloth type.

A central design premise is that goal-conditioned recurrent state-space models (GC-RSSMs) are well matched to the problem. Two reasons are emphasized. First, because the robot never sees the full true cloth state, a latent recurrent model is useful for belief tracking under partial observability. Second, flattening is sequential, so world models support imagination-based planning under long-horizon dynamics. The action primitive is quasi-static high-level PnP rather than fine velocity control, so the representation problem is to encode the evolving cloth configuration compactly and predict the effect of discrete pick-place actions.

2. GC-RSSM formulation and latent dynamics

The base recurrent state-space model follows the standard world-model decomposition into deterministic and stochastic latent states (Kadi et al., 23 Aug 2025). The recurrent latent dynamics are written as

$$\mathbf{h}_t = f(\mathbf{h}_{t-1}, \mathbf{z}_{t-1}, \mathbf{a}_{t-1}),$$

with posterior and prior stochastic states

$$\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t), \qquad \tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t),$$

where $\mathbf{x}_t$ is the observation, $\mathbf{a}_t$ the action, $\mathbf{h}_t$ the deterministic recurrent state, and $\hat{\mathbf{z}}_t,\tilde{\mathbf{z}}_t$ the posterior and prior stochastic states.

LaGarNet extends this into a goal-conditioned RSSM by conditioning both inference and prediction on the goal image $\mathbf{g}$: $$\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t, \mathbf{g}),$$

$$\tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{g}).$$

The stated purpose is to prevent the recurrent model from “forgetting” the target and to make latent rollouts goal-aware.

The model factorization is given as

$$p(\mathbf{x}_{1:T}, \mathbf{z}_{1:T} \mid \mathbf{a}_{1:T}, \mathbf{g}) = \prod_{t=1}^T p(\mathbf{x}_t \mid \mathbf{z}_t)\, p(\mathbf{z}_t \mid \mathbf{z}_{t-1}, \mathbf{a}_{t-1}, \mathbf{g}),$$

$$\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t), \qquad \tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t),$$0

The latent state is intended to represent the hidden physical configuration of the cloth. More specifically, $$\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t), \qquad \tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t),$$1 is the stochastic latent state encoding uncertain cloth configuration, while $$\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t), \qquad \tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t),$$2 is the deterministic recurrent memory that propagates temporal information. Together they act as a learned belief state over garment deformation. The qualitative reconstruction analysis reported in the paper indicates that this latent state is particularly effective at tracking sleeves and garment boundaries over imagined future steps. This suggests that the goal-conditioning does not merely specify a terminal objective; it also shapes the intermediate latent geometry used for planning.

The goal is provided as a goal image $$\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t), \qquad \tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t),$$3, encoded by the same visual encoder used for the current observation. The resulting goal embedding $$\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t), \qquad \tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t),$$4 is injected into the latent inference and transition models. A practical component is goal augmentation: rather than requiring a canonical goal image at test time, the goal input is subjected to step-wise augmentation and Gaussian translation

$$\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t), \qquad \tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t),$$5

This is used to bridge sim-to-real appearance gaps and to avoid requiring a perfectly canonical goal state in real experiments.

3. Objectives, reward design, and data collection

LaGarNet combines world-model learning with reward learning (Kadi et al., 23 Aug 2025). The training objective is stated as maximizing the log-likelihood of observations and rewards conditioned on actions and goal: $$\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t), \qquad \tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t),$$6 Using the RSSM factorization, the loss is written as

$$\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t), \qquad \tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t),$$7

The learned components therefore include observation reconstruction, reward prediction, and KL alignment between posterior and prior latent dynamics. The implementation notes include no gradient stop on goals during training, gradient stop for overshooting loss, and an increased model capacity relative to PlaNet-ClothPick, with deterministic, stochastic, and hidden dimensions raised from $$\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t), \qquad \tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t),$$8 to $$\hat{\mathbf{z}}_t \sim q(\hat{\mathbf{z}} \mid \mathbf{h}_t, \mathbf{x}_t), \qquad \tilde{\mathbf{z}}_t \sim p(\tilde{\mathbf{z}} \mid \mathbf{h}_t),$$9, alongside two extra linear layers in the reward head.

The coverage-alignment reward is a core part of the system. It uses step-wise coverage improvement together with change in Max IoU between current and goal cloth masks. A SpeedFolding-style approximation is defined as

$\mathbf{x}_t$0

with

$\mathbf{x}_t$1

Here, $\mathbf{x}_t$2 is the delta coverage reward and $\mathbf{x}_t$3 is the delta Max IoU reward. The reward is then modified to preserve flattened states and discourage disturbing near-success configurations: $\mathbf{x}_t$4 with

$\mathbf{x}_t$5

The reported motivation is that this reward can be used in both simulation and real-world settings, unlike rewards that require oracle particle distances or special canonicalization.

The dataset collection strategy is treated as a separate novelty. The pipeline is: collect a small number of human demonstrations, train a Diffusion Policy on those demonstrations, and then mix trajectories from that learned policy with those from a mask-biased random policy. The mask-biased random policy samples the pick pixel uniformly from the cloth mask and the place pixel from the whole image, avoiding impossible grasps while preserving broad exploration. In the comparison setup, the diffusion policy is trained with 50 human demonstrations, and the LaGarNet family is trained on about 300k transitional steps from the proposed collection scheme. The paper further reports that the exact ratio between diffusion and random data does not strongly affect NC or Max IoU, but it does affect success rate. This suggests that exploration coverage and data quality play distinct roles in world-model training and downstream planning.

4. Planning pipeline, environments, and evaluation protocol

At planning time, the GC-RSSM is used inside model predictive control (MPC) to choose actions that maximize predicted flattening reward (Kadi et al., 23 Aug 2025). The complete system therefore consists of a learned latent dynamics model, a learned reward predictor, and an online planner operating over imagined rollouts.

The simulation environment is taken from Canberk et al. / ClothFunnels, with extensions for misgrasping and multi-layer grasping. These extensions are introduced because naive parallel grippers can fail in the real world, and the simulation is intended to reflect that. Garments are drawn from the Cloth3D dataset. The common simulation settings are: scale factor 0.8, camera height 2 m, dark background, cloth stiffness parameters stretch = 0.75, bend = 0.02, shear = 0.02, cloth mass 0.5 kg, and evaluation on 30 hard initial states.

The real-world system uses a UR3e single-arm setup with an Intel RealSense D435i camera at 720 × 1280, mounted 0.72 m above the table. The workspace is ring-shaped, with far radius $\mathbf{x}_t$6 and near radius $\mathbf{x}_t$7. To connect training and deployment, the system uses a workspace transfer heuristic mapping the square observation window used during training to the feasible ring-shaped robot workspace. The gripper hardware is also modified by replacing a tweezer-style extension with a 3D-printed cylinder extension to improve gripping heavier garments.

The evaluation compares LaGarNet against ClothFunnels, PlaNet-ClothPick, JA-TN, Diffusion Policy, and human reference performance. The reported metrics are NC (Normalized Coverage), NI (Normalized Improvement), Max IoU, and SR (success rate). In simulation, success is defined by reaching 90% NC and 80% Max IoU. In real-world experiments, two success variants are used: $\mathbf{x}_t$8 and $\mathbf{x}_t$9.

5. Quantitative performance and ablations

The reported quantitative results indicate that LaGarNet is strong in both simulation and real-world settings (Kadi et al., 23 Aug 2025). In simulation, for long-sleeved T-shirts, it significantly outperforms ClothFunnels, JA-TN, Diffusion Policy, and PlaNet-ClothPick. Representative all-garment results at 20 steps are NC ≈ 89.8%, NI ≈ 79.5%, Max IoU ≈ 74.7%, and SR = 18/30. At 30 steps, the corresponding figures are NC ≈ 94.2%, NI ≈ 87.2%, Max IoU ≈ 79.5%, and SR = 23/30. The paper further states that the single-policy LaGarNet obtains above 90% NC across all garment types in simulation.

The same results also delineate the model’s residual weaknesses. In simulation, LaGarNet still struggles to exceed 85% Max IoU, and the paper notes that it appears to overfit somewhat to skirts and dresses, with weaker performance on T-shirts and trousers in the all-garment setting. This indicates that strong coverage does not automatically imply fine geometric alignment.

In the real world, LaGarNet is reported to substantially improve over PlaNet-ClothPick. At 10 steps, the total performance is NC ≈ 83.3%, NI ≈ 70.3%, Max IoU ≈ 75.8%, and $\mathbf{a}_t$0. At 20 steps, it reaches NC ≈ 87.7%, NI ≈ 77.6%, Max IoU ≈ 79.2%, $\mathbf{a}_t$1, and $\mathbf{a}_t$2. By garment type, skirts are described as the easiest and often best-performing category, whereas dresses are the hardest. Human policies still outperform LaGarNet overall, but the paper states that LaGarNet closes a large fraction of the gap.

The ablations are organized around three claims. First, the GC-RSSM matters: the goal-conditioned recurrent model yields better prior reconstructions and better planning than non-goal-conditioned variants. Second, the coverage-alignment reward matters: reward variants from ClothFunnels, Learning2Unfold, or SpeedFolding approximations are weaker. Third, data collection matters: the mask-biased random policy is crucial for success, and the diffusion policy further improves performance when more data is available. Qualitative results align with these ablations: predicted rewards increase over planning iterations, action distributions become more refined, and the GC-RSSM better tracks sleeve structure than other RSSM variants.

The paper’s broader empirical claim is that LaGarNet reaches state-of-the-art levels for single-gripper garment flattening and compares favorably to earlier mesh-based methods such as VCD and MEDOR. A plausible implication is that, for this task class, carefully trained latent world models can approach the performance of more heavily engineered mesh-centric systems without inheriting all of their structural assumptions.

6. Relation to prior approaches, limitations, and naming ambiguity

LaGarNet’s novelty claim is specific: it is not presented as the first world model for cloth in general, but as the first state-space model that works on complex garments rather than simpler fabrics (Kadi et al., 23 Aug 2025). Relative to prior mesh-based systems, the paper claims reduced inductive bias in several forms: no explicit 3D mesh reconstruction at test time, no garment-specific expert oracle policies, no corner-biased sampling, no dependence on oracle particle distances for reward, no explicit handcrafted mesh dynamics, and less reliance on strong canonical shape assumptions. By contrast, methods such as VCD and MEDOR are characterized as relying on mesh reconstruction, explicit cloth topology priors, and garment-specific reasoning.

Several limitations are stated explicitly. LaGarNet struggles with garments that are too soft or exhibit high internal friction; it has difficulty with wide-open trousers/skirt structures and seam-line reasoning; it still struggles to select and untwist slim parts such as sleeves and legs; it can disrupt already nearly flattened cloth; it does not distinguish front from back of garments; and all-garment training can underfit some garment types while overfitting others. Real-world performance remains below human performance. These limitations constrain any interpretation of the method as a complete solution to garment manipulation.

The name itself also requires disambiguation. In (Kadi et al., 23 Aug 2025), LaGarNet refers to the garment-flattening GC-RSSM system described above. A similarly spelled but distinct term is LaguerreNet, a spectral GNN filter based on continuous generalized Laguerre polynomials for heterophily and over-smoothing in graph neural networks (Goksu, 19 Nov 2025). Another unrelated term is LA-SDG, the learn-and-adapt stochastic dual gradient method for online network resource allocation, which is also described in the supplied material as being called “LaGarNet” (Chen et al., 2017). In current usage within the provided sources, however, the standalone title “LaGarNet” designates the garment-flattening model of (Kadi et al., 23 Aug 2025).

Taken together, these properties position LaGarNet as a latent-dynamics, goal-conditioned, model-based framework for single-arm PnP garment flattening. Its defining contribution is the combination of goal-conditioned latent dynamics, a coverage-alignment reward, a general data-collection strategy, and a workspace transfer heuristic, with evidence that this combination is sufficient to make state-space models viable for complex garment manipulation.