Grasp-LLaVA: LLaVA-Inspired Grasping Systems

• Grasp-LLaVA is a family of systems that fuse language and vision to ground robotic grasping through multimodal reasoning.
• Variants range from modular pipelines using bounding box detection to dense affordance mapping and direct numeric pose regression.
• Empirical studies demonstrate enhanced grasp accuracy and speed, while also revealing challenges like occlusion handling and parsing instability.

Grasp-LLaVA denotes a line of LLaVA-derived or LLaVA-inspired systems for language-conditioned robotic grasping. Across the literature, the label is used in several closely related but non-identical senses: as a modular baseline that couples LLaVA with a downstream grasp detector, as a design pattern in which a LLaVA backbone predicts dense affordance masks or conditions a diffusion model, as a reasoning-tuned multimodal LLM that directly emits numerical grasp poses, and, in a later prosthetics setting, as a specific vision-language grasp estimator for grasp-type classification (Jin et al., 2024, Ma et al., 2024, Vuong et al., 2024, Xu et al., 2024, Zandigohar et al., 11 Aug 2025). This suggests that Grasp-LLaVA is best understood as a research family organized around the use of LLaVA-style multimodal reasoning for grasp grounding, pose generation, and grasp-type inference, rather than as a single standardized architecture.

1. Terminology and scope

The earliest explicit Grasp-LLaVA interpretation in the provided literature is the baseline called “LLaVA → GR-ConvNet” in reasoning grasping, where LLaVA localizes a target from language and vision, and GR-ConvNet predicts grasps inside the resulting crop (Jin et al., 2024). Subsequent work broadens the concept. GLOVER presents a direct path to a “Grasp-LLaVA” system by using LLaVA-7B as a multimodal encoder, adding an affordance token <AFF>, decoding dense affordance masks, and coupling them to a geometry-aware planner (Ma et al., 2024). Language-driven Grasp Detection proposes that a Grasp-LLaVA system can be realized by replacing CLIP/BERT + ALBEF with LLaVA while retaining a diffusion-based grasp generator and contrastive objective (Vuong et al., 2024). RT-Grasp is described as “effectively a Grasp-LLaVA-style system” because it fine-tunes LLaVA-7B-v0 to output reasoning tokens followed by numeric grasp poses (Xu et al., 2024). Grasp-HGN then uses the name Grasp-LLaVA for a concrete grasp-type estimator aimed at semantic projection on unseen objects (Zandigohar et al., 11 Aug 2025).

Work	Use of Grasp-LLaVA	Primary output
(Jin et al., 2024)	“LLaVA → GR-ConvNet” modular baseline	Bounding box, then grasp maps
(Ma et al., 2024)	LLaVA-based affordance reasoning path to “Grasp-LLaVA”	Dense affordance mask and AGE pose
(Vuong et al., 2024)	LLaVA replacement for CLIP/BERT + ALBEF in LGD	2D grasp rectangle
(Xu et al., 2024)	“Grasp-LLaVA-style” reasoning-tuned VLM	Planar pose $p=\{x,y,\theta\}$
(Zandigohar et al., 11 Aug 2025)	Named Grasp-LLaVA model in HGN	One-word 13-class grasp label

A common thread across these variants is that language is not merely used for task planning upstream of grasping. Instead, language conditions the grasping representation itself: a bounding box, a pixel-level affordance mask, a diffusion trajectory, a numeric pose sequence, or a semantic grasp class. The main axes of variation are therefore the output space, the degree of geometric coupling, and whether reasoning is used only for localization or directly for action parameterization.

2. The modular baseline: LLaVA for referring, GR-ConvNet for grasping

In “Reasoning Grasping via Multimodal LLM,” the Grasp-LLaVA baseline is a two-stage pipeline. The inputs are an RGB image $v$ and a language instruction $t$. LLaVA-1.6-34B reasons over the image and text and is prompted to produce a 2D bounding box around the referred object or part. GR-ConvNet then receives the crop $v|_B$ and predicts a grasp quality map $q(x,y)$, angle maps parameterized by $\cos(2\theta)$ and $\sin(2\theta)$, and a width map $w(x,y)$. Candidate grasps are selected from the top-$k$ local maxima of the quality map (Jin et al., 2024).

The grasp representation used downstream is the standard rectangle-style parameterization

$g=(x,y,\theta,w,q),$

with angle decoding

$v$0

Evaluation follows the rectangle metric, under which a prediction is valid if $v$1 and $v$2 (Jin et al., 2024).

The benchmark results show that this modular Grasp-LLaVA baseline underperforms badly on implicit and part-level reasoning grasping. Without depth, its object-level explicit performance is $v$3, $v$4, $v$5, while part-level implicit performance drops to $v$6, $v$7, $v$8. With depth, the corresponding part-level implicit scores are still only $v$9, $t$0, $t$1 (Jin et al., 2024). The same study attributes these failures to inaccurate part bounding boxes in clutter, the coarseness of box-level grounding, and the fact that GR-ConvNet no longer has direct access to language once the crop has been produced.

This baseline is historically important because it isolates a central failure mode of early Grasp-LLaVA designs: strong VLM reasoning at the referring stage does not by itself produce strong language-conditioned grasp prediction. The hand-off from a symbolic or coarse spatial intermediate to a grasp detector loses precisely the fine-grained semantic signal required for implicit intention understanding and part-level affordance control.

3. From box selection to dense affordance and diffusion-conditioned grasp generation

A more integrated interpretation of Grasp-LLaVA appears in task-oriented affordance reasoning. GLOVER uses LLaVA-7B as the multimodal encoder, adds an affordance token <AFF>, and follows LISA’s “Embedding-as-Mask” approach to decode continuous affordance masks directly in RGB feature space. The core sequence is

$t$2

where $t$3 is the learned token feature associated with <AFF> (Ma et al., 2024). The LLM is frozen, the affordance decoder is fine-tuned, and the mask supervision uses a sigmoid focal loss over soft affordance labels formed from Gaussian bumps around interaction points. This turns a LLaVA backbone into a pixel-level affordance predictor rather than a box localizer.

GLOVER couples that affordance mask to Affordance-Aware Grasping Estimation (AGE). High-probability pixels are projected to 3D, clustered with DBSCAN, approximated by a superquadric, and aligned with a gripper ellipsoid under distance, normal-alignment, and collision penalties. In 30 table-top real-world scenes, GLOVER reports success rates of $t$4 in part identification and $t$5 in grasping, with speeds approximately $t$6 times faster in affordance reasoning and $t$7 times faster in grasping pose estimation than the previous state-of-the-art (Ma et al., 2024). In this formulation, a Grasp-LLaVA system is no longer “LLaVA plus a separate detector,” but a dense affordance model whose output is already shaped for geometry-aware grasp planning.

A parallel but distinct route is given by “Language-driven Grasp Detection.” There, the grasp is a 2D oriented rectangle

$t$8

and the forward diffusion process is

$t$9

The denoising objective is

$v|_B$0

and the contrastive term is

$v|_B$1

with combined objective

$v|_B$2

The paper proposes that a Grasp-LLaVA system can be built by replacing the CLIP/BERT + ALBEF stack with LLaVA while keeping this diffusion denoiser and contrastive guidance mechanism (Vuong et al., 2024).

That proposal rests on a large synthetic corpus. Grasp-Anything++ contains approximately $v|_B$3 samples, over $v|_B$4 objects, and more than $v|_B$5 grasping instructions, with around $v|_B$6 ground-truth grasps at the object-part level and approximately $v|_B$7 associated masks (Vuong et al., 2024). The reported LGD results on the Base/New split are $v|_B$8 with CLIP and $v|_B$9 with BERT, while removing the contrastive loss drops performance to $q(x,y)$0, and removing prediction of $q(x,y)$1 drops it to $q(x,y)$2 (Vuong et al., 2024). These results frame Grasp-LLaVA as a language-conditioned generative model in which LLaVA supplies richer multimodal guidance for denoising, rather than only a prompt-conditioned region proposal.

4. Reasoning tuning and direct numerical pose prediction

RT-Grasp shifts Grasp-LLaVA toward direct action parameterization. It uses LLaVA-7B-v0 with a CLIP ViT-L/14 vision encoder and introduces “Reasoning Tuning,” in which the model first emits a short rationale $q(x,y)$3 and then emits the numeric pose sequence $q(x,y)$4 (Xu et al., 2024). The planar grasp is parameterized as

$q(x,y)$5

with $q(x,y)$6 normalized by image width and height, and $q(x,y)$7 (Xu et al., 2024).

Training uses ordinary auto-regressive token likelihood over the complete output sequence:

$q(x,y)$8

with the decomposition

$q(x,y)$9

No separate regression head is introduced; numeric pose prediction is learned as text generation. The paper reports two training strategies: projection-only pre-training, where the vision encoder and LLM are frozen and only the projection layer is trainable, and LoRA fine-tuning, where LoRA modules are inserted into all linear layers of the LLM while the base LLM and vision encoder remain frozen. The reported LoRA configuration is rank $\cos(2\theta)$0 and $\cos(2\theta)$1, with batch size $\cos(2\theta)$2 and learning rate $\cos(2\theta)$3 for pre-training or $\cos(2\theta)$4 for LoRA (Xu et al., 2024).

The training data consist of $\cos(2\theta)$5k image-text paired samples derived from the Cornell Grasp dataset, which provides $\cos(2\theta)$6 RGB images and $\cos(2\theta)$7 objects. Objects are manually divided into $\cos(2\theta)$8 categories, and category-specific reasoning templates generated by GPT-3.5 are refined and manually verified (Xu et al., 2024). The answer format concatenates a rationale with textual numbers such as “x=0.52, y=0.41, θ=0.24,” after which parsing and clipping enforce range constraints.

The empirical result is that the reasoning phase improves numerical accuracy. On the Cornell benchmark, LoRA fine-tuning yields $\cos(2\theta)$9 image-wise and $\sin(2\theta)$0 object-wise accuracy for RT-Grasp, compared with $\sin(2\theta)$1 and $\sin(2\theta)$2 for No Reasoning-B, and $\sin(2\theta)$3 and $\sin(2\theta)$4 for No Reasoning-A. The reported effect of reasoning is an improvement of $\sin(2\theta)$5–$\sin(2\theta)$6 over No Reasoning-A and $\sin(2\theta)$7–$\sin(2\theta)$8 over No Reasoning-B across settings (Xu et al., 2024). In real-robot experiments with a 7-DoF Franka Emika Panda, Franka Hand parallel gripper, and one Azure Kinect camera, RT-Grasp LoRA reaches $\sin(2\theta)$9 success $w(x,y)$0 on $w(x,y)$1 unseen household items, versus $w(x,y)$2 $w(x,y)$3 for GR-ConvNet and $w(x,y)$4 $w(x,y)$5 for RT-Grasp pre-training (Xu et al., 2024).

Within Grasp-LLaVA research, RT-Grasp is significant because it eliminates the intermediate affordance or detection head and asks a LLaVA-derived model to output actionable numbers directly. Its central claim is not that LLaVA is naturally a regressor, but that inserting an explicit reasoning phase before numeric prediction makes such outputs materially more accurate and conversationally adaptable.

5. Semantic and dexterous extensions beyond parallel-jaw grasping

SemGrasp extends the language-conditioned grasping paradigm into human-like hand pose generation. It introduces a discrete grasp representation aligned with language, using three tokens—orientation, manner, and refinement—denoted $w(x,y)$6. A hierarchical VQ-VAE decomposes the grasp as

$w(x,y)$7

with final reconstruction

$w(x,y)$8

The language-and-object-conditioned MLLM then models $w(x,y)$9 autoregressively (Li et al., 2024). The backbone uses PointBERT for 3D object encoding and Vicuna-7B fine-tuned via LoRA, with a two-stage training schedule and a CapGrasp dataset of about $k$0k detailed captions and $k$1k diverse grasps (Li et al., 2024). Although SemGrasp does not use the name Grasp-LLaVA as its main title, it is explicitly positioned as compatible with LLaVA-style multimodal alignment and as a way to replace direct regression with interpretable discrete grasp tokens.

Its reported semantic-generation performance shows a clear gain over a BERT-based baseline: P-FID improves from $k$2 to $k$3, GPT-4 score from $k$4 to $k$5, and perceptual score from $k$6 to $k$7, while maintaining comparable PD and improved SD mean (Li et al., 2024). This establishes one major extension of Grasp-LLaVA-like systems: moving from low-DoF parallel-jaw actions to structured human-hand configurations in MANO space.

DexVLG pushes this transition further by explicitly situating itself relative to Grasp-LLaVA-like methods. Its stated target is instruction-aligned functional grasping with dexterous, multi-finger hands under single-view RGBD on tabletop scenes, advancing the vision-language grasping paradigm “popularized by systems like Grasp-LLaVA,” which generally output 6-DoF gripper poses for simple parallel-jaw grippers (He et al., 3 Jul 2025). DexVLG introduces DexGraspNet 3.0 with $k$8 million dexterous grasp poses mapped to semantic parts across $k$9 objects, paired with $g=(x,y,\theta,w,q),$0 million captions. The two splits are Ours-Wrap with $g=(x,y,\theta,w,q),$1k objects, $g=(x,y,\theta,w,q),$2M grasps, $g=(x,y,\theta,w,q),$3M captions and Ours-Pinch with $g=(x,y,\theta,w,q),$4k objects, $g=(x,y,\theta,w,q),$5M grasps, $g=(x,y,\theta,w,q),$6M captions (He et al., 3 Jul 2025).

The model uses a flow-matching-based pose head and physics-informed contact objectives. Its differentiable force-closure energy is

$g=(x,y,\theta,w,q),$7

and the LP-based variant introduces optimized contact magnitudes through

$g=(x,y,\theta,w,q),$8

subject to $g=(x,y,\theta,w,q),$9 and $v$00, with a piecewise definition of $v$01 conditioned on $v$02 and $v$03 (He et al., 3 Jul 2025). A directional regularizer

$v$04

and the composite CuRobo-based regularizer

$v$05

are used to encourage stability and natural contact geometry (He et al., 3 Jul 2025).

The selected evaluation results show the scale of the capability gap relative to simpler Grasp-LLaVA-style methods. For grasp style ablation, SamPart3D wrap achieves Suc $v$06 and PGA $v$07, while pinch achieves Suc $v$08 and PGA $v$09. In the denoising paradigm ablation, FlowMatching on SamPart3D reaches Suc $v$10 and PGA $v$11, compared with DDPM at Suc $v$12, PGA $v$13, and DDIM at Suc $v$14, PGA $v$15 (He et al., 3 Jul 2025). The article’s relation to Grasp-LLaVA is therefore explicit: Grasp-LLaVA marks the simple-gripper VLM baseline regime, while DexVLG marks the transition to part-aware, multi-contact, dexterous grasp synthesis.

6. Semantic projection and prosthetic grasp-type inference

A distinct and later use of the name appears in “Grasp-HGN: Grasping the Unexpected,” where Grasp-LLaVA is a vision-language grasp estimator designed to improve semantic projection, defined as the ability of a model to generalize to unseen object types by inferring the suitable grasp type from physical characteristics such as shape and size (Zandigohar et al., 11 Aug 2025). Here the problem is not 6-DoF pose generation but 13-class grasp taxonomy prediction for prosthetic-hand control.

The architecture follows a standard LLaVA-style layout: a CLIP vision backbone produces image features, a multimodal projector maps them into the Vicuna token space, and a Vicuna LLM performs multimodal attention over projected visual tokens and text tokens. The input image includes a green star superimposed at the gaze point to indicate the target object. During training, a Conversation Generator produces answers of the form: “The image highlights a <object name>. It has a <object shape> shape and <object size> size. Therefore, <grasp type> is the most suitable grasp type to move it. <grasp type> is used when <grasp description>.” At inference, the prompt is reduced to “In one word, tell me the suitable grasp type to grasp the object of interest” (Zandigohar et al., 11 Aug 2025).

The dataset setup is explicitly object-type-disjoint. HANDSv2 contributes $v$16 sampled images from $v$17 videos and $v$18 object classes, MDS-1 contributes $v$19 sampled images from $v$20 videos and $v$21 object classes, and the combined split uses $v$22 seen object classes and $v$23 unseen object classes, with a $v$24 train/validation division inside the seen set (Zandigohar et al., 11 Aug 2025). The output space is a 13-class taxonomy: Large Diameter, Small Diameter, Medium Wrap, Parallel Extension, Distal Type, Tip Pinch, Precision Disk, Precision Sphere, Fixed Hook, Palmar, Lateral, Lateral Tripod, and Writing Tripod (Zandigohar et al., 11 Aug 2025).

The quantitative distinction from earlier Grasp-LLaVA formulations is sharp. On unseen objects, Grasp-LLaVA reaches $v$25 semantic projection accuracy, versus $v$26 for Grasp-CLIP+, $v$27 for Grasp-ViT+, and $v$28 for Grasp-YOLO. On seen objects, Grasp-LLaVA reaches $v$29, while YOLO, Grasp-ViT+, and Grasp-CLIP+ report $v$30, $v$31, and $v$32 respectively (Zandigohar et al., 11 Aug 2025). This is the first provided source in which “Grasp-LLaVA” names the principal model rather than a baseline or architectural template.

The same paper embeds Grasp-LLaVA inside the Hybrid Grasp Network (HGN), where a fast edge model is coupled to cloud Grasp-LLaVA using calibrated confidence. With Dirichlet calibration and threshold $v$33, HGN reaches $v$34 accuracy on unseen objects, a $v$35 improvement over edge-only $v$36, at $v$37 ms average latency and a $v$38 speedup over Grasp-LLaVA alone. In an 80/20 seen/unseen real-world sample mix, HGN with Dirichlet calibration at $v$39 reaches $v$40 average accuracy, a $v$41 gain over edge-only $v$42, while being $v$43 faster than Grasp-LLaVA alone at $v$44 ms and achieving $v$45 of Grasp-LLaVA’s $v$46 accuracy (Zandigohar et al., 11 Aug 2025). In this setting, Grasp-LLaVA functions as a high-accuracy semantic reasoner whose latency can be amortized by calibrated routing.

7. Limitations, misconceptions, and research directions

A recurrent misconception is that Grasp-LLaVA denotes a single canonical model. The surveyed works indicate otherwise. In one setting it is a modular “LLaVA → GR-ConvNet” baseline (Jin et al., 2024); in another it is an affordance-mask architecture built on LLaVA-7B (Ma et al., 2024); in another it is a possible replacement backbone for diffusion-based grasp generation (Vuong et al., 2024); in another it is a reasoning-tuned numerical predictor using LLaVA-7B-v0 (Xu et al., 2024); and in another it is a named grasp-type classifier for semantic projection (Zandigohar et al., 11 Aug 2025). This suggests that the term denotes an architectural family centered on LLaVA-mediated grounding rather than a fixed benchmark object.

The technical bottlenecks are also consistent across works. Coarse spatial intermediates such as bounding boxes degrade part-level grasping under clutter and occlusion (Jin et al., 2024). RGB-only or single-view setups are sensitive to occlusions, thin structures, transparent or reflective materials, and ambiguous language (Ma et al., 2024, Vuong et al., 2024, He et al., 3 Jul 2025). Direct numeric decoding can suffer parsing instability and remains restricted to planar grasps in RT-Grasp (Xu et al., 2024). Semantic taxonomies can contain underrepresented or visually ambiguous classes, as in Medium Wrap, Lateral Tripod, and Writing Tripod in Grasp-HGN (Zandigohar et al., 11 Aug 2025). Dexterous extensions such as DexVLG reduce the gap to human-like grasping, but they inherit sim-to-real issues, heuristic part categorization, and the assumptions of tabletop single-view RGBD (He et al., 3 Jul 2025).

The main research trajectory is therefore one of progressive integration. Early Grasp-LLaVA systems use LLaVA for reasoning and a separate module for grasp execution. Later systems collapse this boundary by making the LLaVA-derived representation directly predict affordance masks, directly condition a diffusion trajectory, directly emit numeric pose tokens, or directly classify semantically appropriate grasp types. The dexterous line then extends the same principle from simple parallel-jaw grasping to multi-contact hand pose generation with part grounding and physics-informed refinement (Ma et al., 2024, Vuong et al., 2024, Xu et al., 2024, He et al., 3 Jul 2025). A plausible implication is that future Grasp-LLaVA systems will increasingly combine dense grounding, structured reasoning, and physically constrained action generation in a single training objective, while retaining the interactive and open-vocabulary properties that motivated the use of LLaVA in the first place.