OVGrasp: Open-Vocabulary Grasping

• OVGrasp is a family of methods that define open-world, open-vocabulary grasping by linking natural language, scene geometry, and robotics.
• It integrates techniques such as language grounding, affordance inference, and geometric grasp planning to effectively handle clutter and ambiguous instructions.
• The frameworks yield actionable insights through performance metrics and applications ranging from autonomous manipulation to assistive exoskeleton control.

OVGrasp denotes a family of open-vocabulary and open-world grasping problems in which a robot or assistive device must map free-form language, visual context, and scene geometry to a grasp action on the correct object or object part. In recent work, the term covers open-world robotic grasping from arbitrary natural-language instructions, open-vocabulary task-oriented grasping via affordance localization, context-aware target disambiguation in 3D scenes, pragmatic dialogue for object selection, and a hierarchical assistive exoskeleton framework explicitly titled “OVGrasp” (Tziafas et al., 2024, Zheng et al., 2024, Tong et al., 25 Nov 2025, Hu et al., 4 Sep 2025).

1. Scope and task formulations

The central problem is not merely to generate a mechanically stable grasp, but to determine what should be grasped and where it should be contacted under open-ended instructions. In open-world robotic grasping, the instruction may refer to object names, attributes, spatial relations, visual relations, semantic relations, multi-hop reasoning, and affordance. The scene may contain clutter, distractors, and multiple instances of the same category. Traditional systems trained on fixed categories or limited language forms struggle when the query involves compositional reasoning, implicit object descriptions, or unseen objects (Tziafas et al., 2024).

Several papers formalize the problem at different levels. OVAL-Grasp defines the input as an RGB-D image $I \in \mathbb{R}^{H \times W \times C}$ and a natural-language task $t$, with the goal of producing an end-effector pose $g \in SE(3)$ that lies on an object part $x$ that enables the task. GaussianGrasper formulates the task as identifying the referred object in a cluttered 3D scene from a language instruction such as “pick up the water cup” or “grasp the hamburger,” then producing candidate 6-DoF grasp poses. PROGrasp instead begins from an intention-oriented utterance $\ell$, a dialogue history $\mathcal{D}$, and an image $\mathcal{I}$, and seeks the target object region $r^* = \langle x_1, y_1, x_2, y_2 \rangle$ before grasp execution (Tong et al., 25 Nov 2025, Zheng et al., 2024, Kang et al., 2023).

A closely related prerequisite is context-aware entity grounding. OVSG constructs an Open-Vocabulary 3D Scene Graph for objects, agents, regions, and relationships so that queries such as “pick up a cup on a kitchen table” or “Tom’s bottle in the laboratory” can be grounded to the correct physical entity. This is directly relevant when multiple visually similar objects are present and category-level detection is insufficient (Chang et al., 2023).

The assistive interpretation of OVGrasp broadens the scope further. In the 2025 exoskeleton framework, the task is to infer whether the user intends to grasp, release, stop, or do nothing in open, cluttered, everyday environments where the objects are not fixed to a small predefined class set and user intent may change dynamically. Here, grasping is embedded in multimodal human-assistive control rather than only autonomous pick-and-place (Hu et al., 4 Sep 2025).

2. Grounding, affordance inference, and language understanding

OWG organizes open-world grasping into three stages: open-ended referring segmentation, grounded grasp planning, and grasp ranking via contact reasoning. The first stage uses a large vision-LLM, specifically GPT-4V-style multimodal prompting with marked images, chain-of-thought reasoning, and self-consistency, to produce a referred object mask or ID. The supplementary details emphasize marked-image prompting, Set-of-Mark style ID placement with minimal overlap, consistent colors for masks and IDs, reference image plus marked image prompting, structured outputs, and in-context examples (Tziafas et al., 2024).

Task-oriented systems place even greater emphasis on part-level affordance grounding. OVAL-Grasp is explicitly modular and training free. It first uses GPT-4o to decompose the visible object into desirable parts and undesirable parts; it then passes those part names to PartGLEE, or VLPart in ablations, to obtain a binary mask and confidence score for each part. GLOVER instead fine-tunes a LISA-7B / LLaVA + SAM-style architecture by introducing an affordance token <AFF> and predicting a continuous affordance probability map from RGB image and prompt. Its prompt format is "<IMG> What part of the [OBJ] should we interact with to [ACT] it?", omitting the action phrase if no action exists. The LLM is frozen, only the affordance decoder is fine-tuned, and the training loss is sigmoid focal loss (Tong et al., 25 Nov 2025, Ma et al., 2024).

Grounding can also be mediated by structured relational reasoning. OVSG builds a scene graph $G_s$ from RGB-D scan $I$, user language input $t$0, and user position input $t$1, and a query graph $t$2 from the free-form query $t$3. Objects in $t$4 are obtained via OVIR-3D with Detic features; query-side object names are encoded by the CLIP text encoder; agents and regions use Sentence-BERT; abstract relations use GloVe; and spatial relations are handled by a custom Spatial Relationship Predictor. Graph matching then re-ranks scene candidates using Likelihood, Jaccard coefficient, or Szymkiewicz-Simpson index (Chang et al., 2023).

PROGrasp extends grounding to pragmatic human-robot communication. Its modules are Visual Grounding, Question Generation, Answer Interpretation, and Object Grasping, all built by finetuning OFA. The initial utterance does not explicitly name the target category; instead, the robot asks natural, unconstrained follow-up questions and performs pragmatic inference by combining how well Visual Grounding matches the visual-dialogue context with how well Answer Interpretation explains the human response. This design is explicitly inspired by Rational Speech Acts style probabilistic pragmatic reasoning (Kang et al., 2023).

Across these formulations, the literature indicates that OVGrasp is not reducible to closed-set object detection. OWG frames grounding as open-ended referring segmentation; OVAL-Grasp and GLOVER frame it as open-vocabulary affordance localization; OVSG introduces typed scene-query graph matching; and PROGrasp models target discovery through dialogue and pragmatic inference. This suggests that language-conditioned grasping depends as much on disambiguation and affordance inference as on grasp mechanics.

3. Geometric representations and grasp selection

The grounding stage must be coupled to a grasp-generation mechanism that respects contact, clutter, and embodiment constraints. OWG performs grounded grasp planning by constructing a top-down orthographic projection from RGB-D data and passing color and reverse-depth heightmaps to a pretrained GR-ConvNet. Because Mask R-CNN provides segmentation masks in the 2D image while GR-ConvNet provides grasps from orthographic projections, the system aligns them via Hungarian matching on projected 3D centroids using 3D Euclidean distance as the cost. The final contact reasoning stage asks the VLM to identify the object category and what constitutes a good grasp, list the grasp IDs likely to collide with nearby objects, and rank the grasp IDs accordingly (Tziafas et al., 2024).

GaussianGrasper shifts the representation into an explicit 3D Gaussian Splatting field initialized from a limited set of RGB-D views from a RealSense D455. It uses Efficient Feature Distillation to store low-dimensional latent features on Gaussian primitives and distill them back into CLIP space through a decoder $t$5 composed of two fully connected layers. After target localization, the system renders depth and normals, reconstructs local geometry, and passes the object point cloud to AnyGrasp for collision-free candidates. The normal-guided grasp module computes the angle between the grasping line and the surface normal at each contact point, sums the two angles, and rejects candidates whose sum is greater than a threshold $t$6 (Zheng et al., 2024).

OVAL-Grasp encodes task-conditioned contact preference directly in a 2D actionable heatmap $t$7. The heatmap is initialized to zero, the whole object segmentation is added with a positive value, desirable part masks are added with positive contributions, undesirable part masks are subtracted with negative contributions, contributions are scaled by confidence, values are scaled into $t$8, and a Gaussian blur with a $t$9 kernel is applied. ContactGraspNet generates candidate grasps $g \in SE(3)$0, and each candidate is scored by a contact score $g \in SE(3)$1 and a z-axis score $g \in SE(3)$2: $g \in SE(3)$3 The top-scoring grasp is executed (Tong et al., 25 Nov 2025).

GLOVER replaces a learned grasp network with Affordance-Aware Grasping Estimation. The predicted 2D affordance is projected into 3D stereo affordance, processed by voxel downsampling and DBSCAN clustering, fit with a superquadric surface $g \in SE(3)$4, and converted into a 7D gripper pose $g \in SE(3)$5 by aligning a gripper ellipsoid $g \in SE(3)$6 under collision-avoidance and safe-grasp constraints. Its central affordance prediction equations are

$g \in SE(3)$7

with training objective

$g \in SE(3)$8

The affordance supervision itself is a 2D Gaussian bump centered at an interaction point: $g \in SE(3)$9 (Ma et al., 2024)

The assistive OVGrasp framework uses a different grasp-selection logic because the control target is a cable-driven soft glove rather than a robot arm. YOLO-World provides open-vocabulary RGB detections. For each detected object, the center pixel $x$0 is sampled in the registered depth frame to obtain $x$1, yielding a node $x$2 in a dynamic graph $x$3. The hand centroid is $x$4, and the nearest object is selected by

$x$5

If the same target persists for $x$6 frames, the system issues $x$7; voice commands transcribed by FunASR can produce $x$8 or $x$9; and the low-level actuator is regulated by a PID law

$\ell$0

(Hu et al., 4 Sep 2025)

4. Representative system families

The recent literature clusters into a small number of recurrent architectural patterns.

System	Core design	Distinctive emphasis
OWG	VLM grounding + GR-ConvNet + contact reasoning	Open-ended referring segmentation in clutter
GaussianGrasper	3D Gaussian Splatting + EFD + AnyGrasp	Fast open-vocabulary 3D querying and scene update
OVAL-Grasp	GPT-4o + PartGLEE + ContactGraspNet	Desirable/undesirable part reasoning
GLOVER	LLaVA/LISA-style affordance decoder + AGE	Continuous affordance in RGB feature space
OVSG	Open-Vocabulary 3D Scene Graph + graph matching	Context-aware entity grounding
PROGrasp	OFA-based VG/Q-gen/A-int + pragmatic inference	Intention-oriented dialogue
OVGrasp	YOLO-World + multimodal decision-maker + PID	Assistive grasp/release intent detection

OWG and GaussianGrasper are most directly concerned with object-level open-world or open-vocabulary grasping in cluttered scenes, but they differ in representation: OWG relies on VLM prompting and 2D-to-orthographic matching, whereas GaussianGrasper builds a language-aligned 3D Gaussian field (Tziafas et al., 2024, Zheng et al., 2024). OVAL-Grasp and GLOVER move the target from “which object?” to “which part of the object should be grasped for the task?”, but they operationalize that question differently: OVAL-Grasp uses LLM-based part decomposition and segmentation-driven heatmaps, while GLOVER fine-tunes a multimodal model to predict a continuous affordance map and then solves grasp pose estimation geometrically (Tong et al., 25 Nov 2025, Ma et al., 2024).

OVSG and PROGrasp address bottlenecks that are upstream of physical grasp synthesis. OVSG concentrates on context-aware grounding in 3D scene graphs, particularly when the target is specified by relation, ownership, or region. PROGrasp addresses intention-oriented utterances such as “I am thirsty” or “My device runs out of battery,” where the target category is not named initially and must be inferred through question asking and answer interpretation (Chang et al., 2023, Kang et al., 2023).

The assistive OVGrasp framework occupies a distinct subdomain. Its objective is not to plan a 6-DoF robotic pick in a tabletop scene, but to trigger grasp assistance, release, and stop for a wearable exoskeleton in egocentric RGB-D scenes. A plausible implication is that the term “OVGrasp” now spans both autonomous robot manipulation and assistive multimodal intent detection, linked by the common requirement of open-vocabulary scene understanding (Hu et al., 4 Sep 2025).

5. Empirical performance and evaluation regimes

OWG evaluates grounding on 173 manually annotated OCID images with query counts of 42 name, 26 attribute, 33 spatial relation, 19 visual relation, 13 semantic relation, 24 multi-hop, and 16 affordance. On cluttered OCID scenes, specialist supervised methods score 8.0 for PolyFormer and 12.1 for SEEM, CLIP-based zero-shot baselines score 25.9 for ReCLIP, 25.7 for RedCircle, and 21.6 for FDVP, GPT-4V with simple set-of-mark prompting scores 36.1, and OWG reaches 70.4 mIoU. Per-query-type results for OWG are 83.3 for name, 80.1 for attribute, 45.7 for spatial relation, 55.4 for visual relation, 78.8 for semantic relation, 90.3 for affordance, and 59.4 for multi-hop (Tziafas et al., 2024).

GaussianGrasper is evaluated in a $\ell$1 tabletop environment with 44 total objects across 10 scenes, 40 graspable objects, 16 RGB-D viewpoints for initialization, and 120 language-guided manipulation trials on 40 objects. For semantic evaluation, LSeg records 26.4 mIoU and 40.6% localization accuracy, LERF* records 41.3 mIoU, 65.1% accuracy, and 40.27 s/query, and GaussianGrasper records 58.2 mIoU, 87.5% accuracy, and 0.22 s/query. For grasping, LSeg + Depth achieves 26.7%, LERF + AnyGrasp 55.8%, GaussianGrasper without normal filter 78.3%, and GaussianGrasper with normal filter 85.0%. In the update experiment, LERF requires 16 viewpoints, 15 GB, and 30 min, whereas GaussianGrasper uses 5 viewpoints, 4 GB, and 1 min (Zheng et al., 2024).

OVAL-Grasp evaluates on 20 household objects with 3 unique tasks for each, using Part Selection Success and Grasp Success. GraspGPT achieves 60.0% Part Selection Success, 56.7% Grasp Success, and 128.67 s; ShapeGrasp achieves 73.3%, 66.7%, and 22.21 s; and OVAL-Grasp achieves 95.0%, 78.3%, and 19.86 s. In 15 cluttered scenes, Part Selection Success is 26.7% for GraspGPT, 46.7% for ShapeGrasp, and 80.0% for OVAL-Grasp. The language-model ablation with PartGLEE fixed gives 33.3% for DeepSeek-R1 (7B), 58.3% for GPT-3.5 Turbo, and 95.0% for GPT-4o; the segmentation-model ablation with GPT-4o fixed gives 78.3% for VLPart and 95.0% for PartGLEE (Tong et al., 25 Nov 2025).

GLOVER is tested in 30 real-world scenes on a UR5e robot arm with an Orbbec Femto Bolt RGB-D camera. The scenes include attributes, relations, complex scenes, tool use, and function reasoning, each evaluated 10 times with varied positions and orientations. GLOVER reports 86.0% affordance reasoning success and 76.3% grasping success, with task-wise results of 95.7% / 84.3% for Attribute, 80.0% / 73.3% for Relation, 82.0% / 68.0% for Complex scene, 88.0% / 80.0% for Tool use, and 76.0% / 68.0% for Function reason. On the AGD20K hard split it reports KLD $\ell$2, SIM $\ell$3, and NSS $\ell$4. The runtime comparison states about 230 s for LERF-TOGO versus about 0.7 s for GLOVER in affordance reasoning, and about 4.0 s for LERF-TOGO with GraspNet versus about 0.1 s for GLOVER AGE in grasp pose estimation (Ma et al., 2024).

OVSG evaluates on ScanNet, DOVE-G, and ICL-NUIM. On ScanNet whole queries, OVIR-3D yields Top1 $\ell$5 of 0.37 and Top1 grounding success of 38.56%, while OVSG-L yields 0.55 and 58.85%. On DOVE-G whole queries, OVIR-3D yields Top1 $\ell$6 of 0.35 and Top1 grounding success of 35.5%, while OVSG-L yields 0.51 and 54.25%. On ICL-NUIM whole queries, OVSG-L reports Top1 $\ell$7 of 0.61, Top1 grounding success of 74.09%, and Top1 $\ell$8 of 0.64. In a real manipulation experiment with a KUKA IIWA 14 and Robotiq 3-finger adaptive gripper, OVSG-L significantly improves success over OVIR-3D when identical blocks must be selected by spatial context (Chang et al., 2023).

PROGrasp reports offline target discovery results of 87% / 79% on validation, 90% / 75% on test-seen, 83% / 61% on test-unseen, and 88% / 42% on test-cluttered for [email protected] / [email protected]. It also reports GPT-4V scores of 29 / 9 / 1 on [email protected] / [email protected] / [email protected], GPT-4V + VG scores of 82 / 82 / 68, and PROGrasp scores of 87 / 87 / 79 in the validation comparison. In online physical experiments on a Kinova Gen3 lite, total object discovery / success are 56 / 33 for SilentGrasp, 82 / 46 for LiteralGrasp, and 84 / 50 for PROGrasp; on ambiguous samples, the figures are 42 / 30 for SilentGrasp and 80 / 53 for PROGrasp (Kang et al., 2023).

The assistive OVGrasp framework is evaluated on 15 objects across 3 grasp types with ten healthy right-handed participants. The abstract reports a grasping ability score of 87.00%, while the detailed results report Overall grasping score $\ell$9, Overall maintaining score $\mathcal{D}$0, and Overall GAS $\mathcal{D}$1. Per grasp type, GAS is $\mathcal{D}$2 for Pinch, $\mathcal{D}$3 for Spherical, and $\mathcal{D}$4 for Cylindrical. The study also reports improved kinematic alignment with natural hand motion and contrasts egocentric against eye-in-hand sensing, with egocentric performing better for low-profile cylindrical objects because the eye-in-hand camera can collide with the table and limits approach angles (Hu et al., 4 Sep 2025).

6. Limitations, failure modes, and research directions

The literature is explicit that open-vocabulary grasping remains bottlenecked by grounding quality, perception robustness, and embodiment constraints. OWG depends on a high-capability proprietary VLM, exhibits occasional non-determinism and parsing issues, is sensitive to visual marker quality in clutter, and can fail by grounding a distractor object, failing to find the target, or identifying the right object but outputting the wrong numeric ID. Its CLIP-based ablations also show that even strong visual prompting combinations such as FGVP* remain below OWG’s 70.4 mIoU, with best reported custom configurations at 51.8 and 51.2 (Tziafas et al., 2024).

GaussianGrasper inherits two major limitations from its 3D sensing stack: a static-scene assumption and difficulty with transparent objects because reliable depth and normal supervision are unavailable. OVAL-Grasp is open-loop, supports only single-step grasps, has no recovery or multi-stage manipulation, depends on pretrained foundation models, and is not real-time. Its remaining failures are mostly due to LLM hallucinating an incorrect decomposition, VLM failure to generate a valid segment, or ContactGraspNet producing only marginal or no candidate grasps (Zheng et al., 2024, Tong et al., 25 Nov 2025).

GLOVER narrows some of the latency issues but remains primarily focused on grasping rather than long-horizon manipulation, and AGE lacks tactile sensing for fragile objects such as glass. OVSG is limited by dependence on OVIR-3D and LLM parsing, and its likelihood aggregation is acknowledged as simplistic. PROGrasp can still fail under occlusion, on unseen objects, or because physical grasping remains difficult even when localization is correct. The assistive OVGrasp framework identifies transparent objects, deformable objects, and clothing-like visual ambiguity as persistent sources of error, and its participant study was conducted on healthy subjects rather than patients with impairment (Ma et al., 2024, Chang et al., 2023, Kang et al., 2023, Hu et al., 4 Sep 2025).

A recurrent misconception is that OVGrasp is only an object-recognition problem. The cited systems instead treat it as a joint problem of language grounding, relational or pragmatic disambiguation, affordance localization, geometric filtering, and control. Another misconception is that a large multimodal model alone is sufficient for robotic deployment. The evidence is mixed: OWG still relies on segmentation and grasp synthesis, PROGrasp shows that GPT-4V is weak at producing tight bounding boxes, and the assistive OVGrasp system suppresses false triggers through spatial proximity, temporal queueing, and symbolic control rather than detector outputs alone (Tziafas et al., 2024, Kang et al., 2023, Hu et al., 4 Sep 2025).

Taken together, these works indicate a convergent research direction. Object-level open-world grounding, part-level affordance reasoning, explicit 3D scene representations, graph-structured context, pragmatic interaction, and multimodal intent detection are increasingly treated as complementary rather than competing solutions. This suggests that future OVGrasp systems are likely to remain hybrid: language-conditioned and open-vocabulary in semantics, but tightly coupled to geometry-aware grasp selection and embodiment-specific control.