CLASP: Clothes Manipulation via Semantic Keypoints

• The paper introduces a semantic keypoint abstraction that bridges high-level planning by vision–language models with low-level dual-arm execution.
• It employs a modular two-stage perception pipeline using off-the-shelf models (OWLV2, SAM, DINOv2, GPT-4o) to extract and label sparse 3D garment landmarks.
• Evaluations demonstrate significant performance gains over baselines in simulation and real-robot experiments across multiple garment types and manipulation tasks.

Searching arXiv for CLASP and directly related clothes manipulation papers. CLothes mAnipulation with Semantic keyPoints (CLASP) is a hierarchical framework for robotic manipulation of deformable garments in which the cloth state is represented by sparse, semantically named keypoints such as “left sleeve,” “right shoulder,” and related garment landmarks. In its 2025 formulation, CLASP is presented as a general-purpose system for multiple clothes types—including T-shirts, shorts, skirts, and long dresses—and multiple tasks, including folding, flattening, hanging, and placing; it uses semantic keypoints as the interface between high-level planning by vision–LLMs and low-level execution by a parameterized dual-arm skill library (Deng et al., 26 Jul 2025). An earlier preprint described the same core idea in a two-level hierarchy that linked semantic keypoint detection from depth imagery to LLM-powered task planning and low-level control, with experiments on T-shirts, trousers, skirts, and towels (Deng et al., 2024).

1. Conceptual basis and problem formulation

CLASP frames clothes manipulation as the problem of producing a robot control policy $\pi$ that, given an RGB-D observation $o$ of a deformable cloth and a language instruction $\ell$, produces a trajectory $\tau$ of arm joint commands such that the cloth reaches a desired configuration (Deng et al., 26 Jul 2025). The motivation is the high-dimensional geometry of cloth: the full cloth state lies in a very high-dimensional manifold $X$, for example the positions of many surface points or particles, which makes direct estimation and planning difficult under deformation and occlusion.

The central abstraction is a semantic keypoint set $K=\{k_1,\dots,k_N\}$, where each keypoint has a position $p(k_i)\in\mathbb{R}^3$ and a text label $s(k_i)$. The resulting state representation is

$s=\{(s(k_i),p(k_i))\}_{i=1}^N .$

The representation is explicitly characterized as sparse, spatial, and semantic: sparse because $N\ll |X|$; spatial because each $o$0 localizes a salient cloth feature; and semantic because labels connect directly to language-based reasoning (Deng et al., 26 Jul 2025).

This formulation treats sleeves, shoulders, collars, hems, and analogous garment landmarks as manipulation affordances. In CLASP, they are both perceptual anchors and action anchors: they can be detected from visual observations and can also serve as grasp or place targets during folding, hanging, and related tasks (Deng et al., 26 Jul 2025). A common misconception is that CLASP attempts to reconstruct the full deformable state. The framework instead adopts a deliberately lossy but task-salient representation, using a small number of named points rather than a dense cloth model.

2. Semantic keypoints as the intermediate representation

The 2025 CLASP paper defines semantic keypoints as the mechanism that bridges high-level task planning and low-level action execution (Deng et al., 26 Jul 2025). Their role is not merely descriptive. Because the keypoints are tied to language labels, a planner can produce symbolic subtasks directly over entities that are grounded in the current scene; because the same keypoints have 3D coordinates, the low-level controller can instantiate those subtasks as gripper motions.

The earlier preprint formalized this with a fixed vocabulary of semantic keypoints for each garment category, for example

$o$1

and a time-varying 3D configuration

$o$2

together with a per-frame heatmap tensor $o$3 whose channels peak at projected keypoint locations (Deng et al., 2024). That formulation makes explicit that CLASP is not restricted to discrete symbolic reasoning; it also supports geometric control through 3D point estimates.

The semantic-keypoint design places CLASP in contrast to two adjacent lines of cloth-manipulation research. One line emphasizes fine-grained semantic regions or edges, such as neckline edges, sleeve cuffs, hems, and grasp regions, using depth-based segmentation rather than point labels (Ren et al., 2021). Another line focuses on grasp-point recognition from RGB-D under unconstrained conditions by combining garment key-part detection and wrinkle analysis, for example necks of shirts or waists of pants, rather than building a multi-task hierarchical representation (Martínez et al., 2017). CLASP adopts a more compact representation than dense segmentation, while preserving garment semantics more explicitly than purely local grasp-point detectors.

3. Perception pipeline and keypoint extraction

The 2025 paper describes a two-stage keypoint extraction pipeline operating on top-down RGB-D images and explicitly states that there is no end-to-end trainable network; instead, the system uses off-the-shelf OWLV2, SAM, DINOv2, and GPT-4o (Deng et al., 26 Jul 2025). In the first stage, “Discovery on Prototype,” a clean prototype image for one exemplar per garment category is processed by OWLV2, SAM, silhouette extraction, contour computation, medial-axis skeletonization, and farthest-point sampling over contour and skeleton candidates. GPT-4o is then prompted with a marked prototype image to select a semantic subset and assign labels. In the second stage, “Matching to Novel Garment,” a test image retrieves its nearest prototype by Best-Buddies Similarity on DINOv2 features with rotations, then performs coarse region proposal via VLM visual prompting and fine matching by

$o$4

followed by 3D recovery through depth back-projection (Deng et al., 26 Jul 2025).

For evaluation, three human experts manually label 120 test images, corresponding to 15 garments and 8 poses, and CLASP compares predicted $o$5 against the expert mean $o$6 (Deng et al., 26 Jul 2025). The paper also gives a hypothetical supervised loss for keypoint training,

$o$7

while emphasizing that the deployed perception stack is not trained end to end.

The earlier preprint described a different perception subsystem centered on a spatio-temporal masked-autoencoder over depth sequences from a top-down Intel RealSense L515 at $o$8 resolution, with $o$9 consecutive frames, ViT-Base encoder, symmetrical ViT decoder, and a keypoint head producing $\ell$0 heatmap channels (Deng et al., 2024). Its training objective combined reconstruction and keypoint supervision:

$\ell$1

and elsewhere in the same description also as $\ell$2 with $\ell$3 (Deng et al., 2024). That earlier version reports detection performance of $\ell$4 px and $\ell$5 on real-world data (Deng et al., 2024).

A concise comparison of the two CLASP descriptions is useful because both are part of the topic’s publication history.

Aspect	Earlier preprint	2025 paper
Perception	Spatio-temporal masked-autoencoder on depth images (Deng et al., 2024)	Prototype discovery and matching with OWLV2, SAM, DINOv2, GPT-4o (Deng et al., 26 Jul 2025)
Planner	LLM-powered task planning (Deng et al., 2024)	VLM task planning with plan verification (Deng et al., 26 Jul 2025)
Real robot	Kinova dual-arm system (Deng et al., 2024)	Franka dual-arm system (Deng et al., 26 Jul 2025)

Taken together, these two versions show a shift from a learned depth-based detector toward a modular foundation-model-based perception stack. A plausible implication is that CLASP’s representation is intended to be stable across different perception back ends, provided that the output remains a semantically labeled set of 3D keypoints.

4. Hierarchical planning and low-level execution

CLASP uses a two-level hierarchy in which a language-conditioned planner operates over semantic keypoints and a low-level controller instantiates the resulting subtasks (Deng et al., 26 Jul 2025). In the 2025 paper, the high-level planner is a multimodal VLM, specifically GPT-4o, prompted in context with an annotated RGB image, semantic keypoint labels, a natural-language instruction, and an available skill set. The prompt asks the model to determine whether the task is already complete and otherwise to propose a sequence of subtasks

$\ell$6

where $\ell$7 is a skill and $\ell$8 refer to semantic keypoints or are empty. The plan is represented as

$\ell$9

The paper further specifies a closed-loop task-planning and verification algorithm that checks task completion, proposes a plan, validates whether each subtask uses admissible skills and known keypoints, and rejects infeasible motions until a valid plan is found or a retry budget is exhausted (Deng et al., 26 Jul 2025).

The low-level manipulation layer in the 2025 system is a pre-built skill library with five primitives: grasp, moveto, release, rotate, and pull (Deng et al., 26 Jul 2025). The grasp primitive can be single-arm or dual-arm; moveto transports held cloth points to target positions; release opens the grippers; rotate rotates cloth about the axis between two grippers by an angle $\tau$0 in the table plane; and pull stretches the cloth by moving the arms apart, with the pull distance chosen as 10% of the current inter-gripper distance. Motion generation is implemented with standard inverse kinematics and a motion planner such as MoveIt!, by solving

$\tau$1

subject to collision and motion constraints (Deng et al., 26 Jul 2025).

The earlier preprint listed a closely related action set,

$\tau$2

and described a Jacobian-based PD controller for low-level execution:

$\tau$3

with a small MLP used to predict pull distance for pull actions (Deng et al., 2024). That version therefore contains an additional press primitive and a more explicitly learned component in the low-level policy.

The system-integration loop in the 2025 paper is iterative: RGB-D input is converted to semantic keypoints; the annotated image and instruction are passed to the VLM planner; the resulting subtasks are executed through low-level skills, inverse kinematics, and motion planning; and after release operations the system re-extracts keypoints before continuing (Deng et al., 26 Jul 2025). The modularity is deliberate. CLASP is not an end-to-end reactive policy; it is a composed architecture in which perception, planning, and execution remain decoupled and interpretable.

5. Simulation benchmarks and quantitative performance

The 2025 paper evaluates CLASP in simulation using SoftGym with cloth represented as 4096 particles, over four garment types—T-shirt, trousers, skirt, and towel—and four tasks each: folding, flattening, hanging, and placing (Deng et al., 26 Jul 2025). The baseline methods are CLIPORT, Goal-conditioned Transporter, FlingBot for flattening only, and FabricFlowNet for folding only. The metric is success rate over 120 trials per task.

For the summary table reported in the paper, the success rates are as follows (Deng et al., 26 Jul 2025):

Method	Seen tasks	Unseen tasks
CLIPORT	Folding 80.0, Flattening 66.7	Folding 0.0, Flattening 9.2
Goal-conditioned Transporter	Folding 76.7, Flattening 85.0	Folding 0.0, Flattening 6.7
FlingBot / FabricFlowNet	Flattening 85.0 / Folding 100.0	Flattening 34.2 / Folding 2.5
CLASP	Folding 95.0, Flattening 80.0	Folding 81.7, Flattening 65.0

The paper explicitly states that CLASP outperforms all baselines on both seen and unseen tasks, including a reported “+81.7% on unseen folding” (Deng et al., 26 Jul 2025). The same section also reports improvements in the task-planning submodule: check-completion accuracy improves from approximately 70% for “VLM only” to approximately 97% for “VLM+KP,” and subtask accuracy improves from approximately 60% to approximately 95% once keypoints are provided (Deng et al., 26 Jul 2025).

The earlier preprint presented a broader simulation breakdown across 30 tasks, again with 120 random trials per task, using flattening coverage $\tau$4, fold error $\tau$5, binary hanging success, and placing success inside a box (Deng et al., 2024). It reported, for example, fold success when $\tau$6 and flatten success when $\tau$7, with representative means such as CLASP at 100.0 on seen T-shirt folding, 83.3 on unseen trousers folding, 96.7 on seen trousers hanging, and 93.3 on unseen T-shirt placing (Deng et al., 2024).

These results establish the intended meaning of “general-purpose” within CLASP: not universal cloth handling, but one framework that covers multiple garment categories and multiple task families without task-specific retraining in the reported setups (Deng et al., 26 Jul 2025).

6. Physical robot implementations

The 2025 paper reports real-robot experiments on a dual-arm Franka system with a RealSense L515, using 15 garments ranging in size from $\tau$8 cm to $\tau$9 cm and including cotton, wool, and silk (Deng et al., 26 Jul 2025). The evaluated tasks are folding, flattening, hanging on a rack, and placing in a box, with 5 trials each. The reported success criteria and outcomes are: folding with IoU $X$0 gives 86% success and an average of 4.3/5; flattening with area $X$1 gives 66% success; hanging with no ground contact gives 94%; and placing with the garment fully in the box gives 92% (Deng et al., 26 Jul 2025). The paper also states that no explicit sim-to-real adaptation was used.

The earlier preprint used a different hardware platform: two Kinova Mico arms, a top-down RealSense L515, and MoveIt! as the motion planner (Deng et al., 2024). That version evaluated 12 real garments, from infant trousers to adult shirts, and reported that no real-world finetuning was used for keypoint transfer, relying instead on mask-and-smooth depth preprocessing with SAM+OWL (Deng et al., 2024). Across four representative tasks—fold, flatten, hang, and place—it reported an average success of approximately 90% over 20 trials each (Deng et al., 2024).

The real-robot studies are important because they delimit what CLASP actually demonstrates. The system has been shown on dual-arm tabletop manipulation of garments under top-down sensing, not on arbitrary full-room laundry scenarios. The reported successes on hanging and placing, particularly 94% and 92% in the 2025 paper, indicate that the semantic-keypoint abstraction can support task families beyond folding alone (Deng et al., 26 Jul 2025).

7. Relation to adjacent work, limitations, and future directions

CLASP belongs to a broader effort to find compact, manipulation-relevant state abstractions for cloth. Earlier work on “Grasp-Oriented Fine-grained Cloth Segmentation without Real Supervision” argued that some tasks require larger semantic regions such as neckline edges, sleeve cuffs, and hems rather than isolated grasp points, and showed that dense depth-based segmentation with multi-layer domain adaptation can detect such regions without real supervision (Ren et al., 2021). A different earlier line, “Recognition of Grasp Points for Clothes Manipulation under unconstrained Conditions,” used active contours, viewpoint feature histograms, and wrinkle analysis to localize garment key parts and infer grasp points on shirts, T-shirts, and pants from RGB-D (Martínez et al., 2017). More recent work on “CloSE: A Compact Shape- and Orientation-Agnostic Cloth State Representation” proposes an analytic representation of corners and fold locations on a unit circle and explicitly discusses semantic labeling and high- and low-level planning for a CLASP-like system (Kamat et al., 7 Apr 2025). Relative to these approaches, CLASP’s defining contribution is the use of sparse, language-grounded semantic keypoints as a shared representation for perception, planning, and execution.

The limitations reported for CLASP are concrete. In the 2025 paper, heavy occlusion can cause keypoint extraction failure when the garment is largely hidden; large garments may exceed the arm workspace; and real-time keypoint tracking is currently too slow, motivating possible distillation to a lightweight CNN (Deng et al., 26 Jul 2025). The earlier preprint likewise notes that low-level actions remain heuristic, dynamic cloth interactions still rely on quasi-static moves, and extending beyond named keypoints to features such as local wrinkles remains open (Deng et al., 2024).

The reported future directions are correspondingly modular. The 2025 paper suggests mobile manipulator integration for large cloths, feedback-driven pull policies based on force for stretch control, and a learned keypoint tracking network for closed-loop pose correction (Deng et al., 26 Jul 2025). The earlier preprint suggests adaptive prompt tuning or few-shot online refinement for planning and exploring more dynamic primitives (Deng et al., 2024). A plausible implication is that CLASP is best understood not as a fixed monolithic algorithm, but as a representational interface: once semantic keypoints are available reliably, different perception stacks, planners, and low-level controllers can be substituted while preserving the same basic hierarchy.