---
title: 'MetaFold: Robotic Garment Folding Framework'
url: https://www.emergentmind.com/topics/metafold
type: topic
---

# MetaFold: Robotic Garment Folding Framework

MetaFold is a modular robotic manipulation framework for garment folding that disentangles task planning from action prediction, enabling robust multi-category folding and language-guided operation. The system leverages point cloud trajectory generation informed by language instructions for high-level planning and employs a low-level foundation model (ManiFoundation) for action synthesis. This architecture facilitates generalization across garment categories, user instructions, and unseen garment instances, achieving state-of-the-art performance in both simulated and real-robot scenarios [2503.08372].

## 1. Architectural Decomposition and System Data Flow

MetaFold divides the garment folding process into two independently optimized modules:

- **High-level planning:** Language-guided point cloud trajectory generation determines the sequence of target garment configurations.
- **Low-level control:** Robot actions are generated by ManiFoundation, a foundation model for contact synthesis operating on pairs of consecutive point clouds.

The data flow is as follows:

| Acquisition Stage          | Input                    | Output                          |
|---------------------------|--------------------------|---------------------------------|
| Real-world (perceptual)    | RGB-D image              | Segmented, downsampled point cloud $\mathcal{P}\in\mathbb{R}^{N\times 3}$ |
| Simulation                 | Mesh vertices            | Downsampled point cloud         |
| High-level planner         | $\mathcal{P}$, $\mathcal{L}$ (language) | Trajectory $\mathcal{T} = \{\mathcal{P}_1,\dots,\mathcal{P}_M\}$ |
| Low-level controller       | $(\mathcal{P}_t, \mathcal{P}_{t+1})$  | Contact actions $\boldsymbol{a} = \{(\boldsymbol{p}_i, \boldsymbol{s}_i)\}$ |
| Execution and Feedback     | -                        | New state $\mathcal{P}'$, closed-loop replanning |

Closed-loop feedback enables replanning after each low-level execution until the garment state $\mathcal{P}$ is approximately equal to the goal configuration $\mathcal{P}_{goal}$.

## 2. Language-Guided Trajectory Generation

### 2.1 Input Encoding

Spatial and semantic instruction representations are constructed as follows:

- **Point clouds:** Downsampled $\mathcal{P}\in\mathbb{R}^{N\times 3}$ (from RGB-D/SAM2 or simulation).
- **Language:** Instruction $\mathcal{L}$ (e.g., "fold left sleeve over back") embedded via LLaMA Instruct model, mean-pooled, projected to $\mathcal{F}_L\in\mathbb{R}^{1\times128}$ through an MLP.

Spatial features $\mathcal{F}_P$ are extracted with PointNet++ to obtain $\mathcal{F}_P\in\mathbb{R}^{N\times128}$.

### 2.2 Conditional Variational Trajectory Model

The core of the planner is a Conditional Variational Autoencoder (CVAE) with transformer-based encoder and decoder:

- **Encoder:** $q_\phi(z\mid \mathcal{F}_P, \mathcal{F}_L, \mathcal{T}_{gt})$, conditioned on features and ground-truth trajectory at training.
- **Latent variable:** $z\in\mathbb{R}^d$ encodes diverse plausible fold trajectories.
- **Decoder:** $p_\theta(\mathcal{T}\mid \mathcal{F}_P, \mathcal{F}_L, z)$, outputs a predicted trajectory.

During inference, latent is sampled $z\sim \mathcal{N}(0,I)$.

Trajectory features $\mathcal{F}_T\in\mathbb{R}^{M\times N\times256}$ are decoded and projected to the predicted trajectory $\widehat{\mathcal{T}}\in\mathbb{R}^{M\times N\times 3}$.

### 2.3 Optimization

Training maximizes the evidence lower bound (ELBO):

\[
\mathcal{L}_{traj} = \mathbb{E}_{z\sim q_\phi}\bigl[\|\widehat{\mathcal{T}} - \mathcal{T}_{gt}\|_2^2\bigr] + \lambda\,\mathrm{KL}\bigl(q_\phi(z\mid\cdot)\,\|\,\mathcal{N}(0,I)\bigr)
\]

This combines per-frame L2 trajectory reconstruction and KL regularization of the latent space.

## 3. Low-Level Foundation Model for Robotic Action

### 3.1 ManiFoundation Model

Action prediction is performed by ManiFoundation, a foundation model for contact synthesis:

- **Input:** Pairs of successive point clouds $(\mathcal{P}, \mathcal{P}')$ and their computed flow.
- **Output:** Contact locations and motion vectors $\{c_i = (\boldsymbol{p}_i, \boldsymbol{s}_i)\}$ with $\boldsymbol{p}_i\in\mathbb{R}^3$ (grasp), $\boldsymbol{s}_i\in\mathbb{R}^3$ (direction and distance).

### 3.2 Fine-Tuning and Ensemble Prediction

- **Dataset:** Simulated folding episodes with known contact labels.
- **Loss:** For each contact,

\[
\mathcal{L}_{contact} = \sum_i \|\boldsymbol{p}_i^{pred} - \boldsymbol{p}_i^{gt}\|_2^2 + \gamma\,\|\boldsymbol{s}_i^{pred} - \boldsymbol{s}_i^{gt}\|_2^2
\]

- **Ensembling:** ManiFoundation is sampled with 160 seeds per step. Outputs are clustered within a small threshold $\varepsilon$ in $\boldsymbol{p}$-space; final action is selected as contact nearest the cluster mean.

### 3.3 Interface

The predicted trajectory $\widehat{\mathcal{T}}$ is decomposed into consecutive pairs, each passed to ManiFoundation to yield the next robot action, closing the high- to low-level loop.

## 4. Empirical Evaluation and Comparative Performance

### 4.1 Datasets

- **MetaFold dataset:** 1,210 meshes; 3,376 folding trials (2,664 train, 712 test); categories: no-sleeve, short-sleeve, long-sleeve, pants.
- **Zero-shot:** 500 previously unseen Cloth3D garments.

### 4.2 Metrics

- **Rectangularity:** $\frac{\text{Area}(\text{folded mesh})}{\text{Area}(\text{min bounding rectangle})}$ (higher better).
- **Area Ratio:** $\frac{\text{Area}(\text{folded})}{\text{Area}(\text{unfolded})}$ (lower, more compact).
- **Success Rate:** Fraction achieving thresholds on above metrics.

### 4.3 Baselines

- **UniGarmentManip:** Dense visual correspondence.
- **GPT-Fabric:** LLM + keypoint detection policy.
- **3D Diffusion Policy:** End-to-end action model.
- **Deng et al.:** Language-guided deformable manipulation.

### 4.4 Results

| Category   | MetaFold Rectangularity | UniG | DP3 | MetaFold Success Rate | UniG  | MetaFold Zero-Shot Success |
|------------|------------------------|------|-----|----------------------|-------|----------------------------|
| No-sleeve      | 0.87                 | 0.85 | 0.85| 0.97                 | 0.90  | 0.97                       |
| Short-sleeve   | 0.83                 | 0.78 | 0.82| 0.88                 | 0.71  | 0.88                       |
| Long-sleeve    | 0.85                 | 0.88 | 0.86| 0.90                 | 0.86  | 0.93                       |
| Pants          | 0.86                 | 0.81 | 0.88| 0.96                 | 0.84  | 0.79                       |

MetaFold achieves higher rectangularity and success rates across all garment types relative to prior methods. For previously unseen garments (Cloth3D), it achieves success rates of $0.79$–$0.97$. In language generalization (unseen instructions), success rates remain $0.63$–$0.93$ versus $0.46$–$0.56$ for prior work.

## 5. Ablation Studies and Qualitative Observations

Ablation experiments confirm the necessity of architectural components:

- **Ours w/o ManiFoundation:** Naïve contact selection reduces success from 0.86 to 0.27.
- **Ours w/o closed-loop:** Open-loop execution lowers success to $\sim$0.07.
- **Reduced frequency planning (5 or 15 frames):** Moderate decline in rectangularity/success.
- **Next-step only (no full trajectory):** Success drops to 0.41.

Qualitative analyses (visualizations of ground-truth and generated folding trajectories) demonstrate coherent behavioral adaptation to different language instructions. Real-robot experiments with an xArm6 and RealSense D435 validate trajectory tracking and generalization across garment categories.

## 6. Generalization, Constraints, and Prospects

MetaFold’s use of raw point clouds (eschewing keypoints or garment templates) allows a single model to fold T-shirts, tank-tops, and pants. The CVAE and LLaMA Instruct encoder provide open-vocabulary instruction grounding, mapping user utterances to canonical fold actions.

Identified constraints include sensitivity to RGB-D segmentation errors, limitation to per-subtask planning (requiring pre-partitioned garment parts), and the stochasticity of ManiFoundation’s sampling (necessitating ensembling). Future research directions include:

- Integrating part-aware segmentation for autonomous decomposition of multi-stage folding,
- End-to-end fine-tuning to couple high- and low-level modules,
- Exploiting stronger vision-language models for free-form instruction following,
- Domain adaptation for sim-to-real transfer without heavy real-robot data requirements.

MetaFold demonstrates that modular disentanglement—trajectory planning conditioned on language and continuous robot contact optimization—enables high generalization and robust real-world performance for complex nonrigid manipulation tasks [2503.08372].

Source: https://www.emergentmind.com/topics/metafold