AdvGrasp: Multimodal Robotic Grasping Overview

• AdvGrasp is a term for multiple robotic grasping frameworks that integrate attribute-based, language-driven, and physics-aware adversarial methods.
• The attribute and language-driven approaches fuse visual and textual features end-to-end, enabling adaptation to novel objects and reliable grasp predictions.
• The adversarial framework perturbs object geometry for robustness evaluation, highlighting the gap between neural grasp scores and physical execution.

AdvGrasp is a name used for multiple distinct robotic grasping systems rather than a single canonical method. In the literature represented here, it denotes an attribute-based instance grasping framework with one-grasp adaptation, a language-driven grasp detection method presented under the name LGD, and a physically grounded adversarial attack framework for robotic grasping. Across these usages, the term is associated with three different problem formulations: multimodal target-conditioned grasp affordance prediction, language-conditioned grasp-pose generation, and mechanics-based robustness evaluation through adversarial object deformation (Yang et al., 2021, Vuong et al., 2024, Wang et al., 14 Jul 2025).

1. Terminological scope and disambiguation

The principal source of confusion around AdvGrasp is terminological rather than technical. The 2021 paper "Attribute-Based Robotic Grasping with One-Grasp Adaptation" uses AdvGrasp to describe an end-to-end multimodal grasping system that grounds textual attributes such as color, shape, and name in cluttered RGB-D scenes. The 2024 paper "Language-driven Grasp Detection" describes AdvGrasp under the method name LGD and frames grasp prediction as conditional diffusion over grasp rectangles given an image and a natural-language instruction. The 2025 paper "AdvGrasp: Adversarial Attacks on Robotic Grasping from a Physical Perspective" uses the same name for an attack framework that deforms object geometry to reduce lift capability and grasp stability (Yang et al., 2021, Vuong et al., 2024, Wang et al., 14 Jul 2025).

Usage of the name	Paper	Core formulation
Attribute-based AdvGrasp	(Yang et al., 2021)	End-to-end multimodal instance grasping with one-grasp adaptation
AdvGrasp/LGD	(Vuong et al., 2024)	Diffusion-based language-driven grasp detection
Physical AdvGrasp	(Wang et al., 14 Jul 2025)	Adversarial object deformation against grasp mechanics

A common misconception is therefore that AdvGrasp refers to one lineage of algorithms. The papers instead show three independent lines of work that share a label while addressing different failure modes and conditioning signals.

2. Attribute-conditioned instance grasping

In "Attribute-Based Robotic Grasping with One-Grasp Adaptation," AdvGrasp addresses attribute-based instance grasping in clutter: a robot observes a top-down RGB-D workspace and receives a text query such as "red sphere" or "apple, red sphere." The task is to predict instance grasping affordances for the specified target rather than for an object class alone. The method assumes color, shape, and name as attribute types, and it is trained self-supervised in simulation using basic objects of various colors and shapes, with random textures and domain randomization over RGB values, object sizes, object heights, and workspace textures (Yang et al., 2021).

The architecture consists of a multimodal encoder and an affordances decoder. The pre-grasp workspace image is orthographically projected into a heightmap $v_{\text{pre}}$. A ResNet-18 visual spatial encoder produces a dense feature map $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$, and a deep averaging network text encoder maps the query $t$ to a 512-dimensional vector $\varphi_t$. Fusion is performed by gated attention,

$$F_{\text{att}} = \varphi_{v,\text{spa}} \odot \text{Broadcast}(\varphi_t),$$

so that visual channels are gated by the query text. To handle grasp orientation, the fused tensor is rotated by $N=6$ discrete orientations, each corresponding to multiples of $30^\circ$, and a fully convolutional residual decoder outputs a grasp affordance map $Q_g$ for top-down parallel-jaw grasps (Yang et al., 2021).

The grasping loss is

$$\mathcal{L}_m = (q_e-\bar{q}_e)^2 + \frac{\lambda_M}{|M|} \sum_{i \in M} q_i^2,$$

where $q_e$ is the predicted score at the executed pixel, $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$0 is the ground-truth label for that action, and $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$1 is a background mask from depth. A second component learns a joint visual-textual attribute space from object persistence: for a successful grasp, the difference between scene embeddings before and after grasping should match the text embedding of the grasped object. The paper formalizes this with a triplet-based multimodal metric loss and combines the two terms as

$$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$2

Hindsight Experience Replay is used during self-supervised data collection, and successful post-grasp scenes $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$3 are stored for metric learning (Yang et al., 2021).

The paper’s distinctive adaptation mechanism is "one-grasp adaptation." At test time, a novel object is placed alone in the workspace, the generic model obtains one successful grasp, that grasp is rotated into multiple orientations, and the model is fine-tuned using only $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$4. The object’s name token is appended to the query text, and the new name embedding is initialized so as not to disturb the existing query vector too much. Reported instance grasping success before adaptation is $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$5 on simulated basic objects, $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$6 on simulated novel objects, and $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$7 on real-world objects; after one-grasp adaptation, performance reaches $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$8 on simulated novel objects and $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$9 on real-world objects (Yang et al., 2021).

3. Language-conditioned diffusion grasp detection

In "Language-driven Grasp Detection," AdvGrasp is presented under the method name LGD and reframes grasp detection as conditional generation. The input is an image $t$0 and a natural-language prompt $t$1, and the target is a 2D grasp rectangle

$t$2

The paper explicitly contrasts this formulation with prior methods such as GR-ConvNet, GG-CNN, Det-Seg-Refine, CLIPORT, and CLIP-Fusion, arguing that earlier approaches either ignore language or use it indirectly in two-stage or fused pipelines. LGD instead predicts the grasp pose end-to-end under language conditioning (Vuong et al., 2024).

The associated dataset, Grasp-Anything++, is central to the formulation. It contains $t$3M samples, over $t$4M objects or object masks, more than $t$5M grasping instructions, and $t$6M ground-truth poses at the object-part level. The construction pipeline uses ChatGPT to generate scene descriptions and grasp instructions, Stable Diffusion to synthesize images, OFA, Segment Anything, and VLPart to ground object and part references, and a pretrained grasp annotator, RAGT-3/3, to generate grasp rectangles. Candidate grasps are filtered using a torque-based quality check derived from antipodal grasp quality, hallucinated images are manually filtered, and prompts are deduplicated with BERTScore at a threshold of $t$7. The dataset spans $t$8 LVIS categories, has scene descriptions averaging about $t$9 words, and contains substantial occlusion diversity (Vuong et al., 2024).

LGD uses a ResNet-50 vision encoder, a text encoder that can be either CLIP or BERT, and ALBEF for vision-language fusion. The ALBEF module produces an attention map interpreted as an estimate $\varphi_t$0 of the relevant grasp region. Reverse diffusion then denoises from a noisy grasp state $\varphi_t$1 to the final pose $\varphi_t$2, conditioned on image, language, timestep, and noisy grasp state. The architecture is organized into vision encoding, text encoding, ALBEF fusion, timestep embedding, noisy-state embedding, and a final MLP for denoised grasp prediction. What distinguishes the method is the contrastive training objective, which the paper treats as an explicit part of the denoising process rather than as an auxiliary regularizer. Its theoretical support is summarized in Proposition 1: under independence assumptions on $\varphi_t$3, $\varphi_t$4, and noise $\varphi_t$5, and a minimum-distance condition, small contrastive loss implies small estimation error,

$\varphi_t$6

The appendix relates this to the standard diffusion relation

$\varphi_t$7

This suggests that semantic grounding and denoising accuracy are trained in a coupled manner rather than in separate stages (Vuong et al., 2024).

Evaluation uses a base/new zero-shot protocol over LVIS categories, with $\varphi_t$8 of labels treated as seen and $\varphi_t$9 as unseen. Success follows the standard rectangle grasp criterion: IoU above $$F_{\text{att}} = \varphi_{v,\text{spa}} \odot \text{Broadcast}(\varphi_t),$$0 and angle error below $$F_{\text{att}} = \varphi_{v,\text{spa}} \odot \text{Broadcast}(\varphi_t),$$1. On Grasp-Anything++, LGD with CLIP text encoding achieves $$F_{\text{att}} = \varphi_{v,\text{spa}} \odot \text{Broadcast}(\varphi_t),$$2 seen, $$F_{\text{att}} = \varphi_{v,\text{spa}} \odot \text{Broadcast}(\varphi_t),$$3 unseen, and $$F_{\text{att}} = \varphi_{v,\text{spa}} \odot \text{Broadcast}(\varphi_t),$$4 harmonic mean, compared with $$F_{\text{att}} = \varphi_{v,\text{spa}} \odot \text{Broadcast}(\varphi_t),$$5 for CLIP-Fusion. The BERT variant reaches $$F_{\text{att}} = \varphi_{v,\text{spa}} \odot \text{Broadcast}(\varphi_t),$$6. In real-world robotic experiments using a KUKA LBR iiwa, a RealSense D435i camera, and a Robotiq 2F-85 gripper, the method reaches $$F_{\text{att}} = \varphi_{v,\text{spa}} \odot \text{Broadcast}(\varphi_t),$$7 in single-object scenes and $$F_{\text{att}} = \varphi_{v,\text{spa}} \odot \text{Broadcast}(\varphi_t),$$8 in cluttered scenes on $$F_{\text{att}} = \varphi_{v,\text{spa}} \odot \text{Broadcast}(\varphi_t),$$9 daily objects. The paper emphasizes that the model is trained only on synthetic data generated through foundation models yet still transfers reasonably to real scenes, and it also reports cross-dataset generalization and grasp detection "in the wild" on internet images (Vuong et al., 2024).

4. Physically grounded adversarial attacks

"AdvGrasp: Adversarial Attacks on Robotic Grasping from a Physical Perspective" defines AdvGrasp as an adversarial attack framework that targets grasp execution rather than only grasp-quality prediction. The paper argues that attacks on grasp-quality networks are insufficient for three reasons: a grasping system may not rely on a neural predictor at all; a prediction attack does not necessarily imply physical failure; and grasping success depends fundamentally on forces, torques, friction, center of mass, and wrench balance. AdvGrasp therefore attacks two physical metrics corresponding to concrete failure modes: lift capability and grasp stability (Wang et al., 14 Jul 2025).

The object is represented as a triangle mesh $N=6$0 with vertices $N=6$1 and faces $N=6$2. A grasp is

$N=6$3

where $N=6$4 are contact points and $N=6$5 are finger forces. Each contact force is decomposed as $N=6$6 under the Coulomb friction constraint

$N=6$7

Lift capability $N=6$8 measures whether the grasp can overcome gravity and lift the object. The paper defines

$N=6$9

AdvGrasp attacks $30^\circ$0 by deforming the object so that the center of mass shifts,

$30^\circ$1

thereby increasing gravity-induced torque at the contacts (Wang et al., 14 Jul 2025).

Grasp stability $30^\circ$2 measures resistance to external disturbances. If $30^\circ$3 denotes the wrenches available at contact $30^\circ$4, the grasp wrench space is $30^\circ$5, and the stability score is defined as

$30^\circ$6

AdvGrasp attacks $30^\circ$7 by perturbing object regions near the contact points, changing local surface normals $30^\circ$8, altering the contact wrench cone, and shrinking the stability margin in wrench space. The unified optimization objective is

$30^\circ$9

with $Q_g$0 and $Q_g$1. The Laplacian regularizer encourages realistic, smooth deformations and prevents overly jagged shapes (Wang et al., 14 Jul 2025).

The attack pipeline assumes a fixed grasp configuration and does not search for a new grasp. It builds an axis-aligned bounding box around the object, places control points on the AABB surfaces, perturbs them, and deforms the object mesh using interpolation. The paper mentions Mean Value Coordinates-style deformation and reports implementation with a cage/grid strategy. The process is iteratively refined with higher control-point resolution at each stage, under constraints that the object remain geometrically plausible, smooth, physically realizable, and only subtly changed so that the attack is not visually obvious (Wang et al., 14 Jul 2025).

For standardized evaluation, the paper introduces AdvGrasp-20, a benchmark of $Q_g$2 object categories from GraspNet-1Billion. For each object it generates $Q_g$3 grasps for a two-finger gripper with Dex-Net 2.0, $Q_g$4 grasps for a two-finger gripper with GraspNet, and $Q_g$5 grasps for a three-finger gripper with GenDexGrasp, followed by PyBullet validation with a Robotiq 2F-85 and a Robotiq three-finger gripper. Simulation uses soft finger contact with $Q_g$6, torsional friction coefficient $Q_g$7, initial cage size $Q_g$8, perturbation scale $Q_g$9 cage size, simulated annealing with $$\mathcal{L}_m = (q_e-\bar{q}_e)^2 + \frac{\lambda_M}{|M|} \sum_{i \in M} q_i^2,$$0, $$\mathcal{L}_m = (q_e-\bar{q}_e)^2 + \frac{\lambda_M}{|M|} \sum_{i \in M} q_i^2,$$1, cooling rate $$\mathcal{L}_m = (q_e-\bar{q}_e)^2 + \frac{\lambda_M}{|M|} \sum_{i \in M} q_i^2,$$2, and $$\mathcal{L}_m = (q_e-\bar{q}_e)^2 + \frac{\lambda_M}{|M|} \sum_{i \in M} q_i^2,$$3 total cycles. Physical robustness is measured by MinGF, MaxLM, and MaxED. The paper reports that AdvGrasp generally increases MinGF and decreases MaxLM relative to Origin, substantially reduces MaxED, and maintains low Chamfer distance, Hausdorff distance, and $$\mathcal{L}_m = (q_e-\bar{q}_e)^2 + \frac{\lambda_M}{|M|} \sum_{i \in M} q_i^2,$$4 vertex displacement. It also shows that PointNetGPD can label many adversarial grasps as "good" even when they fail physically, supporting the claim that network scores are not sufficient for robustness evaluation. In real-world tests on 3D-printed dabao sod and tomato soup can objects loaded to $$\mathcal{L}_m = (q_e-\bar{q}_e)^2 + \frac{\lambda_M}{|M|} \sum_{i \in M} q_i^2,$$5 kg with lead balls, the original objects can be lifted successfully at the chosen force settings, while the adversarial versions fail: dabao sod under $$\mathcal{L}_m = (q_e-\bar{q}_e)^2 + \frac{\lambda_M}{|M|} \sum_{i \in M} q_i^2,$$6 N and tomato soup can under $$\mathcal{L}_m = (q_e-\bar{q}_e)^2 + \frac{\lambda_M}{|M|} \sum_{i \in M} q_i^2,$$7 N (Wang et al., 14 Jul 2025).

5. Position within broader grasping research

The different AdvGrasp usages are best understood against a broader shift in robotic grasping away from single-gripper, language-agnostic, static-pose formulations. AdaGrasp, for example, learns a single gripper-aware grasping policy that conditions on the geometry of an input gripper and generalizes to novel grippers without retraining. It encodes the gripper from URDF-derived geometry in open and closed states, encodes the scene as a TSDF volume, and uses cross convolution to match gripper features against scene features over translations and orientations. In simulation, it is trained on $$\mathcal{L}_m = (q_e-\bar{q}_e)^2 + \frac{\lambda_M}{|M|} \sum_{i \in M} q_i^2,$$8 training objects and tested on $$\mathcal{L}_m = (q_e-\bar{q}_e)^2 + \frac{\lambda_M}{|M|} \sum_{i \in M} q_i^2,$$9 novel objects, with $q_e$0 training grippers and $q_e$1 test grippers, and it reaches about $q_e$2 to $q_e$3 success in single-object scenes, about $q_e$4 to $q_e$5 in multi-object scenes, and about $q_e$6 to $q_e$7 in clutter with obstacles. In real-world transfer to a UR5 with an Intel RealSense D415, it reports around $q_e$8 success for single-object scenes and $q_e$9 for multi-object scenes (Xu et al., 2020).

D-Grasp extends grasping into dynamic interaction synthesis. Given a static grasp label $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$00, the current object pose $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$01, and a target object pose $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$02, it generates motions that approach, grasp, and move an object to the target 6D pose. The method uses hierarchical reinforcement learning with a low-level grasping policy and a high-level motion synthesis stage running concurrently during the motion phase, all within RaiSim and a MANO-based hand model. The paper reports best motion synthesis performance of $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$03 MPE and $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$04 geodesic error, and its full ablated grasping system achieves success $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$05, SimDist $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$06, and contact ratio $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$07 (Christen et al., 2021).

GraspARL, by contrast, treats dynamic grasping as a "move-and-grasp" adversarial game between a robot and a mover that tries to evade capture. It uses a two-agent Markov game, object-geometry-aware reward design, relative pose state representations, and an adversarial training process in which the mover learns trajectories automatically instead of relying on handcrafted line, sine, or circle motions. The paper reports that GraspARL outperforms UKF+CP, Vanilla RL, and DGRMA across unseen trajectories and speed ratios, and in real-world deployment on a Franka Panda it achieves $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$08 success on potted meat can and $$\varphi_{v,\text{spa}} \in \mathbb{R}^{H \times W \times 512}$$09 on mustard bottle (Wu et al., 2022).

Taken together, these neighboring systems show that the various AdvGrasp usages occupy different points in a larger design space: conditioning on semantics, conditioning on gripper morphology, maintaining grasp stability through dynamics, and evaluating failure through explicit mechanics.

6. Conceptual significance and unresolved issues

The three main uses of the name AdvGrasp reveal different answers to the same underlying question: what additional structure should condition robotic grasping beyond local image geometry? The attribute-based version conditions grasping on text-specified object attributes and supports one-grasp adaptation (Yang et al., 2021). The LGD version conditions grasp generation on natural-language instructions and ties semantic grounding to denoising through a contrastive diffusion objective (Vuong et al., 2024). The adversarial-attack version conditions robustness analysis on mechanics, explicitly targeting gravitational torque and wrench-space stability rather than only prediction scores (Wang et al., 14 Jul 2025).

This also clarifies several recurring misconceptions. One is that semantic grounding and grasp prediction can be cleanly separated into object grounding followed by grasp synthesis; the language-driven work argues that such decomposition can increase inference time and complicate deployment, and instead directly predicts the grasp pose end-to-end (Vuong et al., 2024). Another is that a high neural grasp-quality score implies robust physical execution; the physical attack paper shows that PointNetGPD can rate many adversarial grasps as good even when they fail physically (Wang et al., 14 Jul 2025). A third is that target-driven generalization necessarily requires large real-world supervision; the attribute-based work learns from basic simulated objects with self-supervision and then adapts from a single successful grasp (Yang et al., 2021).

Open issues are also method-specific. The attribute-based system leaves texture, pose, and function as future attributes beyond color, shape, and name (Yang et al., 2021). The physical adversarial framework explicitly notes future work on dynamic settings and defense design (Wang et al., 14 Jul 2025). A plausible implication is that future grasping systems may combine the semantic conditioning of attribute- and language-driven approaches with the mechanics-aware robustness criteria emphasized by physical AdvGrasp, while also incorporating the gripper-awareness and dynamic control strategies exemplified by AdaGrasp, D-Grasp, and GraspARL (Xu et al., 2020, Christen et al., 2021, Wu et al., 2022).