UniDiffGrasp: Unified Open-Vocabulary Grasping

• The paper introduces a unified framework that couples vision–language parsing with constrained diffusion to generate precise 6-DoF grasps on functionally critical parts.
• It demonstrates significant improvements in grasp success and collision-free rates by guiding the sampling process with part-specific geometric constraints.
• The approach extends to dual-arm coordination by independently generating and filtering candidate grasps, enabling robust cooperative manipulation without retraining.

UniDiffGrasp is a unified framework for open-vocabulary, task-oriented grasping that combines vision–language reasoning, open-vocabulary semantic grounding, and constrained diffusion-based grasp generation to produce precise 6-DoF grasps on specific functional parts, and extends the same mechanism to dual-arm coordinated manipulation. In the formulation reported under the title “UniDiffGrasp: A Unified Framework Integrating VLM Reasoning and VLM-Guided Part Diffusion for Open-Vocabulary Constrained Grasping with Dual Arms,” the system parses free-form user instructions into an object, a target part or parts, and an operation mode, grounds these semantics in RGB-D observations, and then uses Constrained Grasp Diffusion Fields (CGDF) with Part-Guided Diffusion to concentrate grasp synthesis on the grounded part geometry without retraining for new constraints. Its dual-arm variant defines two target regions, generates per-arm constrained grasp candidates, and selects a stable cooperative pair through collision and force-closure filtering together with a spacing heuristic (Guo et al., 11 May 2025).

1. Problem setting and operational scope

UniDiffGrasp addresses open-vocabulary constrained grasping, a setting in which the intended grasp is specified not merely by object identity but by a functional region expressed in natural language. The target may be semantically localized, such as a pot’s handles, and the specification may also imply a manipulation mode, including single-arm or dual-arm operation. The central problem is therefore not only to generate stable 6-DoF grasps, but to do so on the correct part under semantic and geometric constraints.

The paper motivates this design by noting that task-oriented grasping requires placing the gripper on parts that afford downstream actions, while ordinary grasp generators tend to produce valid grasps anywhere on the object rather than on the functionally critical region. In the open-vocabulary regime, this difficulty compounds: the system must interpret diverse language, identify the correct object and functional part, and localize that part on the observed geometry. Post-filtering unconstrained samples is described as sample-inefficient, especially when constraints are tight. For dual-arm tasks, the problem becomes more structured still, since two complementary target regions must be defined and coordinated grasps must satisfy local and global constraints simultaneously (Guo et al., 11 May 2025).

This problem formulation places UniDiffGrasp at the intersection of language grounding, geometric perception, generative grasp synthesis, and cooperative manipulation. Its unifying claim is not that one model directly solves all of these subproblems end to end, but that a single semantic-to-geometry-to-grasp pipeline can couple them tightly enough to support precise part-constrained grasping in both single-arm and dual-arm settings.

2. Semantic grounding and system composition

The pipeline begins with a vision–LLM that receives natural language $L$ and an RGB scene image $I$, and predicts the tuple

$\{O, p^*, \mathrm{Mode}\} = \mathrm{VLM}(L, I),$

where $O$ denotes the target object, $p^*$ the target part, and $\mathrm{Mode}$ the operation mode. In the reported implementation, the VLM is GPT-4o.

After semantic parsing, UniDiffGrasp performs hierarchical open-vocabulary grounding. GroundedSAM localizes the object $O$ and returns an object mask $M_a$, from which a global object point cloud $P$ is reconstructed from depth:

$M_a = \mathrm{GroundedSAM}(I, O).$

VLPart is then applied to the masked object view to segment the requested part $I$0, yielding a part mask $I$1 and the target part point cloud $I$2:

$I$3

This object-to-part sequence is important because it constrains part parsing to the already localized object and is reported to mitigate false positives from distractors while providing precise part geometry for constrained grasp generation (Guo et al., 11 May 2025).

The resulting system is modular but not loosely coupled. The part point cloud is not used merely for post hoc scoring; it directly enters the grasp generator as a geometric constraint. That distinction separates UniDiffGrasp from pipelines in which language reasoning and grasp generation are sequential but effectively independent. A common misunderstanding is to treat the VLM as the core grasping component. In UniDiffGrasp, the VLM determines semantic targets, but geometric precision is delegated to the diffusion-based grasp generator conditioned on the grounded part.

3. Constrained diffusion field and part-guided grasp synthesis

UniDiffGrasp represents a 6-DoF grasp by $I$4, with rotation $I$5 and translation $I$6, so that $I$7. Its grasp generator is CGDF, built on SE(3)-DiffusionFields, which define a learned energy function $I$8 over grasp pose, denoising step index, and object geometry. The associated score is

$I$9

UniDiffGrasp does not retrain this diffusion model for new part constraints. Instead, it applies Part-Guided Diffusion at inference by evaluating the same pretrained energy on both the full object point cloud $\{O, p^*, \mathrm{Mode}\} = \mathrm{VLM}(L, I),$0 and the target part point cloud $\{O, p^*, \mathrm{Mode}\} = \mathrm{VLM}(L, I),$1. At denoising step $\{O, p^*, \mathrm{Mode}\} = \mathrm{VLM}(L, I),$2, the global and part energies are

$\{O, p^*, \mathrm{Mode}\} = \mathrm{VLM}(L, I),$3

The guided energy is defined as

$\{O, p^*, \mathrm{Mode}\} = \mathrm{VLM}(L, I),$4

and the guided score becomes a piecewise rule:

$\{O, p^*, \mathrm{Mode}\} = \mathrm{VLM}(L, I),$5

Operationally, the reverse dynamics are steered by whichever constraint is currently more violated. The paper characterizes this as concentrating samples on the target part while preserving global validity, with collision avoidance and stability embodied in the learned energy (Guo et al., 11 May 2025).

A generic manifold update is written as

$\{O, p^*, \mathrm{Mode}\} = \mathrm{VLM}(L, I),$6

where $\{O, p^*, \mathrm{Mode}\} = \mathrm{VLM}(L, I),$7 is a step size, $\{O, p^*, \mathrm{Mode}\} = \mathrm{VLM}(L, I),$8 a noise scale, $\{O, p^*, \mathrm{Mode}\} = \mathrm{VLM}(L, I),$9, and $O$0 denotes pose composition. The final single-arm grasp is selected by minimum global energy at the terminal step:

$O$1

Two features of this formulation are central. First, the part constraint is enforced during sampling rather than after sampling, which the paper presents as a major efficiency advantage over unconstrained generation followed by filtering. Second, the model reuses a pretrained unconstrained CGDF and introduces constraints through inference-time guidance rather than through additional supervised datasets for each constraint class.

4. Dual-arm extension and cooperative grasp selection

For dual-arm tasks, UniDiffGrasp generalizes the same constrained diffusion mechanism by defining two target regions, $O$2 and $O$3. These may be obtained semantically, when the part decomposition naturally yields two functional subparts such as two handles, or geometrically, when a single large region or object must be split into two non-overlapping halves. The system then runs Part-Guided Diffusion independently for each arm using $O$4 and $O$5.

After independent generation, each arm’s candidate set is filtered using thresholds on both global and regional energies:

$O$6

$O$7

The Cartesian product of filtered candidates is then pruned by removing inter-gripper collisions and applying a force-closure stability test following the reference cited as [34] in the paper. Among the remaining stable pairs, the selected grasp pair is the one maximizing the Euclidean distance between gripper centers projected on the object surface:

$O$8

This selection criterion is explicitly heuristic: it encourages balanced, wide-baseline support for stable transport. The paper reports that this strategy is effective for objects with semantically distinct dual contact regions, such as a pot, and also for more symmetric objects such as a basin or keyboard (Guo et al., 11 May 2025).

An important technical clarification is that UniDiffGrasp’s dual-arm component is not a joint two-arm diffusion model over a combined bimanual configuration space. The reported system performs constrained generation independently per arm and then conducts cooperative filtering and selection. This architectural choice simplifies the integration of semantic region constraints, but it also delineates the current scope of the method.

5. Empirical evaluation and reported performance

The reported implementation uses GPT-4o for reasoning, GroundedSAM for object segmentation, VLPart for part segmentation, and a pretrained CGDF for energy-based diffusion on $O$9. The sensing and execution stack comprises a RealSense D435 at $p^*$0 RGB-D resolution, a Baxter dual-arm robot with parallel-jaw grippers, and dual NVIDIA RTX 3090 GPUs. After grasp selection, standard inverse kinematics and motion planning are used for execution (Guo et al., 11 May 2025).

The evaluation uses two principal metrics. Grasp Success Rate (GSR) measures physical success, defined as lift plus stable hold. Collision-Free Rate (CFR) measures the absence of gripper–object mesh collision. The single-arm evaluation compares UniDiffGrasp against GroundedSAM + GraspNet and ThinkGrasp, the latter described as VLPart + GraspNet. Seven everyday objects are tested with 30 trials per method and object. In this setting, UniDiffGrasp achieves average GSR $p^*$1 and CFR $p^*$2.

The dual-arm evaluation considers basin, keyboard, laptop, and pot, again with 30 trials per object and method. The baseline is a dual-arm region-sampling method using FPS+KNN similar to CGDF’s baseline, but without semantic splitting and with simpler selection. UniDiffGrasp achieves average GSR $p^*$3 and CFR $p^*$4, compared with baseline GSR $p^*$5 and CFR $p^*$6.

Setting	Protocol	Reported result
Single-arm part-constrained grasping	7 objects; 30 trials per method/object	GSR $p^*$7, CFR $p^*$8
Dual-arm large-object grasping	4 objects; 30 trials per object/method	GSR $p^*$9, CFR $\mathrm{Mode}$0
Dual-arm baseline	Same dual-arm protocol	GSR $\mathrm{Mode}$1, CFR $\mathrm{Mode}$2

The paper attributes especially large gains to objects with complex part geometry, including pans, mugs, kettles, and semantically meaningful bimanual grasps such as pot handles. Residual failures are traced to segmentation errors under occlusion or clutter, occasional VLM misinterpretation of ambiguous part terms, and rare kinematic or planning failures when optimal grasps lie near joint limits or induce collisions. These observations indicate that the limiting factors in the current system are not restricted to grasp synthesis alone; they include semantic grounding fidelity and downstream motion feasibility.

6. Position within diffusion-based grasping research

UniDiffGrasp occupies a specific niche within diffusion-based robotic grasp generation. Its distinguishing feature is the integration of open-vocabulary reasoning and segmentation with inference-time part-constrained grasp diffusion, together with a dual-arm cooperative extension. In contrast, “Grasp Diffusion Network” learns grasp generation from partial point clouds in $\mathrm{Mode}$3 and augments inference with collision-avoidance cost guidance, but it is framed around single-view parallel-jaw grasp generation rather than open-vocabulary part grounding or dual-arm coordination (Carvalho et al., 2024). “DexGrasp-Diffusion” presents a unified diffusion-based functional grasp synthesis method across multiple dexterous robotic hands, using post hoc physics and functional discriminators, including open-vocabulary affordance selection, but affordance instructions do not directly condition diffusion sampling and the method is centered on multi-hand dexterous grasp synthesis rather than VLM-grounded part-constrained generation in real dual-arm scenes (Zhang et al., 2024). “DexGrasp Anything” integrates differentiable geometric physics terms into both training and sampling for universal dexterous grasping at large scale, but its emphasis is universal physics-aware dexterous grasp generation rather than semantic part grounding and cooperative bimanual selection (Zhong et al., 11 Mar 2025). “Simultaneous Pick and Place Detection by Combining SE(3) Diffusion Models with Differential Kinematics” embodies a different sense of unified diffusion-based grasp planning: it constrains denoising with multi-target differential inverse kinematics so that pick and place are jointly feasible and reachable without in-hand manipulation, addressing executability and placement rather than open-vocabulary part selection (Ko et al., 28 Apr 2025).

Within this landscape, UniDiffGrasp’s contribution is not a new diffusion backbone alone. Rather, it demonstrates how a pretrained energy-based grasp model can be repurposed at inference time for arbitrary part constraints produced by a grounding stack. This makes its notion of “unified” primarily semantic and systems-level: language reasoning, perception, constrained sampling, and dual-arm coordination are assembled into one operational framework. A plausible implication is that the paper’s most consequential idea lies in the conversion of semantic part descriptions into geometric constraints that directly enter the generative process, rather than in replacing the underlying grasp diffusion model itself.

7. Limitations, misconceptions, and prospective directions

The reported limitations are concentrated in four areas. First, part semantics can be ambiguous: colloquial or underspecified terms can lead to grounding errors. Second, while the learned energy implicitly captures stability and collision, fine-grained manipulation constraints such as approach vectors, contact normal distributions, or downstream task objectives like “pourable” orientation are not modeled explicitly. Third, bimanual scalability remains limited for more complex cooperative tasks such as handoffs, re-grasps, or in-hand manipulation, which may require joint optimization of two-arm trajectories with contact switching and explicit object dynamics. Fourth, perception robustness degrades in heavy clutter and partial views, motivating stronger multi-view fusion and uncertainty-aware segmentation (Guo et al., 11 May 2025).

Several common misconceptions follow from the system’s breadth. UniDiffGrasp does not eliminate the need for geometric grounding; it depends critically on GroundedSAM and VLPart. It does not require retraining to accommodate new object-part constraints, but this flexibility is obtained by reusing a pretrained CGDF with different $\mathrm{Mode}$4 inputs, not by making grasp generation language-native in isolation. It also does not solve dual-arm grasping through a monolithic bimanual diffusion state space; cooperative behavior is induced through region definition, per-arm constrained generation, and downstream pairwise filtering.

These limitations also clarify the method’s research significance. UniDiffGrasp suggests that open-vocabulary manipulation can profit from treating language as a constraint-specification mechanism and diffusion as a geometry-respecting solver. This suggests a broader design principle for task-oriented grasping: semantic reasoning is most effective when translated into explicit geometric conditions that shape the sampling dynamics themselves, rather than merely ranking unconstrained grasp proposals after the fact.