GRASPED: Multi-Domain Grasp State Modeling

• GRASPED is a multifaceted term defining grasped-object segmentation, self-supervised instance segmentation, and tactile state detection in robotics, as well as a spectral autoencoder in graph anomaly detection.
• In robotics, GRASPED converts grasp events into robust supervision signals by integrating kinematics, depth, and tactile feedback for improved segmentation, pose estimation, and manipulation control.
• In graph learning, GRASPED employs spectral encoding and polynomial-based decoding to identify node anomalies through unsupervised reconstruction of graph attributes.

GRASPED is a polysemous term in contemporary arXiv literature. In robotics, it appears as a label for grasped-object segmentation, grasped-object-based self-supervision, and a visuotactile state indicating that a cloth edge has been successfully grasped. In graph machine learning, it denotes “Graph Autoencoder with Spectral Encoder and Spectral Decoder,” an unsupervised model for node anomaly detection. Across the robotics usages, a common technical thread is the treatment of a grasped object as an informative state variable rather than as an unmodeled by-product of grasp execution (Florence et al., 2019, Liu et al., 2023, Sunil et al., 2022, Choong et al., 21 Aug 2025).

1. Terminology and scope

In robotics, GRASPED most often refers to perception or control problems centered on a held object. One usage is explicitly definitional: in “Robot-Supervised Learning for Object Segmentation,” “GRASPED” means grasped-object segmentation in the wild, and the paper states that it does not introduce a dataset or benchmark named GRASPED (Florence et al., 2019). A second usage is procedural: “Self-Supervised Instance Segmentation by Grasping” uses grasp interactions to generate segmentation supervision for a downstream instance segmentation model (Liu et al., 2023). A third usage is state-based: “Visuotactile Affordances for Cloth Manipulation with Local Control” operationalizes “GRASPED” as the thresholded tactile decision that an edge has been grasped, enabling subsequent local control (Sunil et al., 2022).

This plurality matters because the term does not identify a single method family. In one literature it names a segmentation problem, in another a self-supervised data-generation pipeline, in another a tactile state for manipulation, and in a separate graph-learning literature it is an acronym for a spectral autoencoder (Choong et al., 21 Aug 2025). A plausible implication is that “GRASPED” should be read contextually rather than as a stable benchmark label.

2. GRASPED as grasped-object segmentation in the wild

“Robot-Supervised Learning for Object Segmentation” formulates grasped-object segmentation as two sub-problems: separating “robot manipulator + object” from the background, and then separating the object from the manipulator (Florence et al., 2019). The first stage is Kinematics-based Foreground Segmentation (KIFS). It over-segments the depth image with Felzenszwalb’s graph-based method, projects 3D manipulator link coordinates into the image, marks plausible depth segments as foreground, and refines the result with GrabCut. The resulting binary mask is $M_{fg}$. The second stage is a Self-Recognition Network (SRN), implemented as Mask R-CNN with an R-50-FPN backbone pretrained on COCO, trained without human labels on manipulator-only images automatically labeled by KIFS.

The central compositional relation is

$$M_{obj} = M_{fg} \setminus M_{robot} = M_{fg} \cap \neg M_{robot},$$

followed by a final morphological opening with $J_3$. KIFS uses a depth plausibility threshold $\lambda = 200$ mm, GrabCut for 8 iterations, and morphological kernels $J_8$, $J_{10}$, and $J_{75}$ in opening, erosion, and dilation. The SRN training set contains 297 manipulator images, augmented with 1,800 backgrounds and 80 foreground object cutouts, yielding 11,988 images in the combined setting (Florence et al., 2019).

Quantitatively, the method reports grasped-object mask performance of $0.639$ mIoU versus $0.362$ for the Browatzki et al. baseline, with precision/recall $0.823/0.716$. It also improves downstream object re-recognition in context: segmentation AP increases from $$M_{obj} = M_{fg} \setminus M_{robot} = M_{fg} \cap \neg M_{robot},$$0 to $$M_{obj} = M_{fg} \setminus M_{robot} = M_{fg} \cap \neg M_{robot},$$1, and box AP from $$M_{obj} = M_{fg} \setminus M_{robot} = M_{fg} \cap \neg M_{robot},$$2 to $$M_{obj} = M_{fg} \setminus M_{robot} = M_{fg} \cap \neg M_{robot},$$3 when Mask R-CNN is fine-tuned on automatically labeled grasped-object images (Florence et al., 2019). The significance of this result is not only segmentation quality in-hand, but the conversion of robot embodiment—forward kinematics, camera calibration, and depth sensing—into a self-supervision signal.

3. GRASPED as self-supervised instance segmentation by grasping

“Self-Supervised Instance Segmentation by Grasping” generalizes the idea of grasp-based supervision from in-hand segmentation to large-scale instance-segmentation training data generation (Liu et al., 2023). Its grasp segmentation model takes as input

$$M_{obj} = M_{fg} \setminus M_{robot} = M_{fg} \cap \neg M_{robot},$$4

where $$M_{obj} = M_{fg} \setminus M_{robot} = M_{fg} \cap \neg M_{robot},$$5 is the grasp mask of the active suction cups, and predicts a visible mask $$M_{obj} = M_{fg} \setminus M_{robot} = M_{fg} \cap \neg M_{robot},$$6 and an amodal mask $$M_{obj} = M_{fg} \setminus M_{robot} = M_{fg} \cap \neg M_{robot},$$7. The architecture uses a ResNet-50 backbone with FPN, two gated residual upsampling blocks, and a final 2-channel head. Training uses pixelwise binary cross entropy with boundary-aware weighting.

The method then applies the trained model to 110k unlabeled grasp pairs, filters outputs using suction-gauge success, occlusion ratio, entropy threshold, and connected-component constraints, and converts the extracted object cutouts into synthetic training images by cut-and-paste composition. Each image may receive 0–25 pasted objects with random rotation $$M_{obj} = M_{fg} \setminus M_{robot} = M_{fg} \cap \neg M_{robot},$$8, scale $$M_{obj} = M_{fg} \setminus M_{robot} = M_{fg} \cap \neg M_{robot},$$9, and random position. To reduce paste artifacts, the pipeline dilates each pasted boundary by 5 pixels and applies Stable Diffusion inpainting for 4 denoising steps on the boundary region (Liu et al., 2023).

The learned grasp segmentation model substantially outperforms image subtraction. With filtering, image subtraction yields mIOU $J_3$0, error $J_3$1, recall $J_3$2, whereas grasp segmentation trained from 100 labeled grasp pairs yields mIOU $J_3$3, error $J_3$4, recall $J_3$5, and training from 200 labeled pairs yields mIOU $J_3$6, error $J_3$7, recall $J_3$8 (Liu et al., 2023). On the downstream instance segmentation task, the best synthetic regime—learned grasp masks plus diffusion inpainting—achieves AP $J_3$9, exceeding the $\lambda = 200$0 AP of a model trained with 1000 labeled images. On the real robot, the same regime reduces grasp error rate to $\lambda = 200$1, compared with $\lambda = 200$2 for the subtraction-based supervision baseline (Liu et al., 2023).

In this usage, GRASPED is less a grasp-state detector than a self-supervised data engine. The grasp event functions as a causal intervention that reveals object identity through scene change.

4. GRASPED as a visuotactile grasp state

In cloth manipulation, GRASPED is explicitly turned into a measurable state. “Visuotactile Affordances for Cloth Manipulation with Local Control” first predicts visually where an edge grasp is likely to succeed, then confirms the outcome with a multi-frame tactile classifier, and finally exploits the confirmed state with tactile pose control (Sunil et al., 2022). The tactile classifier operates on 5 stacked tactile depth frames and predicts five categories: edge, corner, fold, all fabric, and no fabric. The binary supervision view is

$\lambda = 200$3

and an edge is declared “GRASPED” when $\lambda = 200$4.

The classifier reaches $\lambda = 200$5 overall five-way accuracy and $\lambda = 200$6 non-edge accuracy. Once an edge is confirmed, a second tactile network estimates edge center $\lambda = 200$7 and orientation $\lambda = 200$8 in the sensor plane with mean absolute error $\lambda = 200$9 mm for center position and $J_8$0 for orientation, running at $J_8$1 Hz. Local control then regulates the edge in the gripper while sliding to the adjacent corner. The best visuotactile affordance regime, “Sim2Real + Fine-tune (Ours),” achieves precision@k $J_8$2, average attempts $J_8$3, and $J_8$4 edge-grasp success on the real robot (Sunil et al., 2022).

Related thin-material work extends this post-grasp state estimation from binary edge possession to layer counting. “Dynamic Layer Detection of a Thin Materials using DenseTact Optical Tactile Sensors” classifies whether $J_8$5, $J_8$6, $J_8$7, or $J_8$8 layers are grasped after an anthropomorphic rubbing motion, using optical flow, 6-axis wrench, and joint states in a multimodal transformer; it reports $J_8$9 test accuracy on cloth and $J_{10}$0 on paper (Dhawan et al., 2024). This suggests a broader interpretation of GRASPED in manipulation research: not merely that contact exists, but that the post-grasp state can itself be estimated and used to gate downstream actions.

5. Embodying the grasped object for estimation and control

A separate line of work treats the grasped object as an extension of the robot’s body schema. “Multi-Modal Learning of Keypoint Predictive Models for Visual Object Manipulation” learns a multimodal keypoint detector by fusing vision with proprioception and then regresses virtual joint translations $J_{10}$1 that append the grasped object to the robot’s forward kinematic chain (Bechtle et al., 2020). The virtual-joint regression objective is

$J_{10}$2

and the resulting extended kinematic model is used in a differentiable visual MPC loop. In placing tasks, this structured model attains approximately $J_{10}$3 px RMSE in simulation and approximately $J_{10}$4 px on hardware, whereas neural-dynamics baselines range from approximately $J_{10}$5 to $J_{10}$6 px and degrade markedly in long-horizon prediction (Bechtle et al., 2020).

“3D Foundation Models Enable Simultaneous Geometry and Pose Estimation of Grasped Objects” addresses the same embodiment problem from monocular RGB, using DUSt3R to obtain unscaled geometry and relative poses and then solving for a global scale $J_{10}$7 and fixed object-to-gripper transform $J_{10}$8 without camera extrinsic calibration (Zhi et al., 2024). The method optimizes a multi-view reprojection loss over $J_{10}$9 and $J_{75}$0, and then defines object-anchored forward kinematics through $J_{75}$1. On six objects, the structured rendering-based method reports symmetrized pixel distances such as hammer $J_{75}$2, screwdriver $J_{75}$3, block $J_{75}$4, and brush $J_{75}$5, far below the unstructured direct-regression baseline (Zhi et al., 2024).

With soft tactile interfaces, embodiment must include sensor mechanics. “Manipulation via Membranes” learns membrane dynamics from tactile depth maps, reaction wrenches, and actions, then recovers object pose with an ICP-based observation model and uses MPPI for control (Oller et al., 2022). On unseen tools, it reports drawing score $J_{75}$6 and pivoting error $J_{75}$7, outperforming linear tactile dynamics and direct object-pose dynamics baselines. The same embodied-object viewpoint also appears in vibro-tactile work: “Extended Tactile Perception” infers contact information transmitted through grasped rods and tools, obtaining localization errors less than $J_{75}$8 cm on a $J_{75}$9 cm rod and food-identification accuracy $0.639$0 with NUSkin and SVM-RBF plus EST features (Taunyazov et al., 2021).

Taken together, these works suggest that in advanced manipulation, a grasped object is increasingly modeled as part of the robot’s latent state, geometric state, or tactile transmission path rather than as an external disturbance.

6. GRASPED in graph anomaly detection

Outside robotics, GRASPED denotes a graph-learning model: “Graph Autoencoder with Spectral Encoder and Spectral Decoder” for node anomaly detection (Choong et al., 21 Aug 2025). The method is explicitly unsupervised and is motivated by the observation that anomalies on graphs induce spectral shifts, including a right-shift of energy toward higher graph frequencies. Its encoder is based on Graph Wavelet Convolution, parameterized as a linear combination of Haar scaling functions across multiple decomposition levels, and its attribute decoder is a Wiener Graph Deconvolution network implemented by polynomial approximation.

The overall training objective combines degree reconstruction, neighborhood-distribution reconstruction, and attribute reconstruction,

$0.639$1

and the node anomaly score is

$0.639$2

The decoder uses a Wiener solution

$0.639$3

and approximates spectral filters by Remez polynomials for scalability (Choong et al., 21 Aug 2025).

On real-world graph anomaly detection datasets, the model reports ROC-AUC values of $0.639$4 on Reddit, $0.639$5 on Disney, $0.639$6 on Books, $0.639$7 on Enron, and $0.639$8 on Weibo (Choong et al., 21 Aug 2025). This usage is semantically unrelated to robot grasping except for the shared name, and it is therefore the clearest example of GRASPED’s cross-domain polysemy.

7. Common themes, misconceptions, and extensions

One recurrent misconception is that GRASPED names a single robotics benchmark. The segmentation literature states the opposite: “GRASPED” there refers to grasped-object segmentation, not to a dataset or benchmark (Florence et al., 2019). A second misconception is that GRASPED always denotes successful grasp completion. In fact, different papers assign it different ontological roles: a segmentation problem (Florence et al., 2019), a self-supervision mechanism (Liu et al., 2023), a thresholded tactile state (Sunil et al., 2022), a post-grasp layer-count classifier (Dhawan et al., 2024), or an acronym in graph anomaly detection (Choong et al., 21 Aug 2025).

The forward-looking directions are likewise heterogeneous. For embodied kinematic modeling, proposed extensions include articulated objects, rotational virtual joints, online slip detection, and closed-loop replanning (Bechtle et al., 2020). For robot-supervised segmentation, proposed extensions include multi-view fusion, depth-aware SRN inputs, temporal consistency, and calibration self-checks (Florence et al., 2019). For cloth manipulation, the stated directions include broader datasets, tactile regrasp, dynamic manipulation during sliding, and tighter integration of global visual priors with local tactile states (Sunil et al., 2022). For dynamic layer detection, proposed future work includes continuous thickness estimation, broader material coverage, and closed-loop adaptation of normal load and shear (Dhawan et al., 2024).

Across these literatures, the most stable meaning of GRASPED is not lexical but methodological: it marks situations in which the fact of being grasped becomes a source of structure, supervision, or state information. In robotics, this often means converting embodiment, contact, or tactile response into a model that is useful for segmentation, pose recovery, or control. In graph learning, it denotes a spectral autoencoder whose name is historically independent of grasping but equally centered on reconstruction as a route to anomaly identification (Choong et al., 21 Aug 2025).