VCoT-GraspSet: Dataset for Visual CoT Grasping

• VCoT-GraspSet is a large-scale grasp dataset designed for language-driven prediction by integrating explicit visual chain-of-thought annotations for object localization.
• It comprises 167,000 synthetic images with 1.36M grasp annotations and over 400 real-world images, covering 388 diverse object categories.
• Experiments show that incorporating intermediate detection steps improves grasp success by 15–18% in cluttered and real-world environments.

VCoT-GraspSet is a large-scale, refined grasping dataset specifically introduced to enable the training and evaluation of language-driven robotic grasp generation models that incorporate visual chain-of-thought reasoning for improved interpretability and generalization in cluttered and realistic environments. It is tightly associated with the VCoT-Grasp framework, which addresses the challenges of multi-turn visual and linguistic reasoning for robust, language-guided grasp prediction in both synthetic and real-world scenarios (Zhang et al., 7 Oct 2025).

1. Purpose and Definition

The VCoT-GraspSet is constructed to support end-to-end grasp foundation models that require both perceptual input (RGB images) and explicit intermediate reasoning steps (object localization via bounding boxes). It enables the realization and benchmarking of visual chain-of-thought (VCoT) grasping models that must handle:

• Multi-object clutter
• Distractor objects and varying backgrounds
• Zero/few-shot transfer to novel object categories
• Interpretable reasoning traces via intermediate detection steps

This dataset differs from prior grasp sets in its focus on supporting chain-of-thought reasoning via explicit intermediate annotation (bounding boxes), high-fidelity synthetic and real domains, and language-driven task specification.

2. Dataset Composition and Annotation Protocol

The VCoT-GraspSet comprises:

• 167,000 synthetic RGB images, each annotated with over 1.36 million grasp poses
• More than 388 object categories, including 367 in the training/testing splits and 21 reserved for zero/few-shot generalization
• 400+ high-resolution real-world images with over 1,200 grasp annotations

Each data sample includes:

• The input RGB image
• One or more rectangular grasp annotations, each parameterized as $g = [x, y, w, h, \theta]$, where $x, y$ is the grasp center in image coordinates, $w, h$ are the dimensions, and $\theta$ is the orientation in $[0^\circ, 180^\circ]$
• Intermediate target object bounding boxes, serving as the visual chain-of-thought context for stepwise reasoning about “where” before “how” to grasp

Annotation quality is enforced through multi-stage processing: automated open-vocabulary detection (using YOLO-World), filtering by Intersection-over-Union (IoU) thresholds, and further manual validation via crowdsourcing.

3. Visual Chain-of-Thought Structure

The chain-of-thought paradigm embedded in VCoT-GraspSet reflects the two-stage processing logic of the corresponding model:

1. Target Localization: Given a language instruction, the system predicts the bounding box $b$ of the referred object within the image ($b = \pi(O, l_d)$, with $l_d$ the detection instruction).
2. Refined Grasp Generation: The predicted bounding box crops a high-resolution region $O_b$ from the image, serving as the focus for generating the final grasp rectangle $g$ based on $O, O_b, l_g$ (where $l_g$ is the grasp instruction).

These intermediates create interpretable reasoning traces and facilitate robust performance in cluttered or ambiguous contexts.

4. Data Generation, Refinement, and Validation

VCoT-GraspSet is generated through a systematic pipeline:

• Synthetic Data: Large-scale synthetic scenes are rendered with randomized object arrangements, backgrounds, and lighting.
• Annotation Processing: Grasp annotations are refined by aligning synthetic labels with bounding boxes inferred by open-vocabulary detectors (YOLO-World).
• Quality Assurance: Annotations with poor localization (low IoU) are discarded. The remaining set undergoes manual validation via a crowdsourcing interface, improving overall annotation reliability.
• Real-World Data: A dedicated subset of over 400 high-resolution images with more than 1,200 meticulously curated grasps enables evaluation under real sensory conditions and out-of-distribution backgrounds.

This protocol reduces label noise endemic to synthetic datasets and ensures that both intermediate detections and grasps are accurate.

5. Evaluation Protocols and Experimental Outcomes

The VCoT-GraspSet underpins rigorous evaluations of grasp foundation models using multiple metrics:

• Prediction accuracy for grasp rectangles (measured via IoU and orientation, in the LM-head variant reaching 83.60% on seen objects)
• Bounding box localization as an explicit intermediate step
• Generalization to unseen categories, validated over the held-out 21-category set
• Simulation-to-reality transfer, measured through grasp success rates on physical robotic platforms

Experiments demonstrate that models trained on VCoT-GraspSet, especially with explicit chain-of-thought (CoT) reasoning, exhibit significantly higher success rates on unseen objects and in the presence of distractors compared to models lacking such reasoning intermediates. For example, VCoT-Grasp with an LM head attains around 71% grasp success on unseen objects, outperforming baselines by a margin of 15–18 percentage points (Zhang et al., 7 Oct 2025).

6. Significance and Comparative Context

Relative to prior datasets, VCoT-GraspSet introduces several substantive innovations:

Dataset	# Images (synthetic/real)	CoT Annotation	# Object Categories	Realism Refinement
Cornell	~885 (real)	No	280+	Minimal
VMRD	~5,000 (real)	No	31	Moderate
GraspNet-1Billion	97,280 (synthetic/real)	No	190	Real robot eval
VCoT-GraspSet	167,000/400+	Yes	388	Crowdsourced

VCoT-GraspSet is the first to systematically combine large-scale synthetic coverage, intermediate chain-of-thought annotation, fine-grained category structure, and real-world evaluation splits tailored for visual-linguistic grasp reasoning.

7. Applications and Implications for Robotic Grasp Synthesis

VCoT-GraspSet directly enables training and evaluation of language-driven grasp foundation models leveraging visual chain-of-thought reasoning—a requirement for interpretable robot behavior in ambiguous, cluttered, or open-world settings. The dataset supports:

• Multi-turn reasoning architectures that decompose object localization and grasp synthesis
• Benchmarks for generalization to novel categories and backgrounds
• Evaluation of robustness in zero/few-shot and OOD scenarios

A plausible implication is that similar dataset structures—combining large-scale synthetic coverage with validated reasoning intermediates—will become essential for scalable, foundation-model-centric robotic learning frameworks moving forward.

Summary

VCoT-GraspSet defines a new high-water mark for grasp dataset design, integrating large-scale multistage synthetic and real-world imagery, explicit chain-of-thought annotations, and rigorous refinement and validation protocols. It provides critical infrastructure for benchmarking and advancing the performance and interpretability of language-driven visual-grasping models, underscoring the importance of intermediate visual reasoning for robust robotic manipulation in open and cluttered domains (Zhang et al., 7 Oct 2025).