Embodied-Points-200K: Dataset for Embodied AI

• Embodied-Points-200K is a large-scale dataset that operationalizes pointing as an embodiment-agnostic intermediate representation for embodied AI by integrating diverse spatial tasks.
• It compiles ~200K verified ‘question–verification’ samples from heterogeneous sources including visual reasoning, manipulation logs, simulation, part annotations, and video trajectories.
• The dataset supports training and evaluating Vision-Language Models for robust spatial grounding, region localization, and motion tracing across both simulated and real-world settings.

Embodied-Points-200K is a large-scale, purpose-built dataset that operationalizes pointing as an embodiment-agnostic intermediate representation for embodied AI and robotics. Constructed to support and evaluate four core embodied pointing abilities in Vision-LLMs (VLMs), it provides approximately 200,000 “question–verification” samples spanning diverse domains such as general visual reasoning, physical manipulation, functional part recognition, and motion tracing. Embodied-Points-200K is the foundational training corpus for the Embodied-R1 VLM, facilitating robust transfer across simulated and real-world embodiment settings, and is designed to help address the perception-action gap in robotics by unifying spatial language grounding, region localization, affordance tracking, and object-centric manipulation traces (Yuan et al., 19 Aug 2025).

1. Dataset Composition and Domains

Embodied-Points-200K integrates heterogeneous sources, resulting in a corpus of ~$2\times10^5$ samples. The key data domains and their contributing datasets include:

• General visual reasoning: RefCOCO, featuring natural images and referring expressions.
• Embodied robotics benchmarks: RoboRefIt and RoboPoint datasets, providing cluttered tabletop scenes and manipulation tasks.
• Open-source manipulation logs: ≈1 million raw episodes filtered to ~33,000 high-quality samples for two-dimensional (2D) region grounding.
• Simulation: Isaac Gym–based synthetic 3D rearrangement episodes (≈10,028).
• Functional part annotations: HandAL dataset, covering 212 real objects and yielding ~40,000 grasp-point annotations.
• Video corpora for motion tracing: Internet and robotics sources for trajectory generation tasks.

This diversity allows the dataset to cover a wide range of spatial relations, object types, manipulation primitives, and scene geometries, supporting broad generalization.

2. Four Core Embodied Pointing Abilities

Embodied-Points-200K is explicitly annotated to supervise four foundational pointing modalities:

1. Referring Expression Grounding (REG):
   • Input: Image and referring natural-language instruction (e.g., "the red mug handle").
   • Target: Single 2D pixel coordinate $p\in [0,w]\times [0,h]$ that must lie within the corresponding instance mask.
2. Region Referring Grounding (RRG):
   • Input: Image and relational placement instruction (e.g., "place the block between the cup and the bowl").
   • Target: Single 2D free-space coordinate, or—if 3D variant—$(X,Y,D)$ in camera coordinates, denoting placement in the workspace.
3. Object Functional Grounding (OFG):
   • Input: Image and affordance-focused query ("where to grasp the knife to cut").
   • Target: 2D coordinate inside a functionally salient part, verified against HandAL's part-bounding boxes.
4. Visual Trace Generation (VTG):
   • Input: Instruction indicating object movement or trace specification.
   • Target: Ordered sequence $\tau = \{p_1,...,p_8\}$ of eight 2D coordinates forming a manipulation trajectory.

Each modality is annotated through automated or semi-supervised conversion of existing mask, bounding box, or part-annotation labels, typically by centroid or uniform sampling inside valid regions. For VTG, a pipeline involving object proposal, keypoint extraction, trajectory tracking, and spline smoothing is used to derive continuous traces from video data.

3. Annotation Pipeline and Data Processing

The collection, conversion, and quality assurance procedures are modality-specific:

• REG: Aggregates mask-based referring expressions from RefCOCO, RoboRefIt, and RoboPoint. Converts masks to centroids or sample points within the mask region; no manual annotation is added.
• RRG (2D/3D): Extracts object placement regions from manipulation logs, computes geometric relations, back-projects onto the image plane, and optionally records 3D coordinates. Isaac Gym simulation augments coverage for 3D tasks.
• OFG: Leverages HandAL’s part polygons, which are converted to tight bounding boxes. GPT-4o generates affordance-focused questions, and points are sampled within affordance regions.
• VTG: Applies object-centric keypoint proposal, trajectory tracking (CoTracker3), cubic spline fitting, and downsampling. Quality filters ensure only feasible, correctly scaled trajectories are included.

Quality assurance combines heuristic filters (e.g., region size, occlusion checks) and iterative manual spot-checks. Expert verification achieves >95% point-in-mask accuracy for REG/OFG, and >90% trajectory-feasibility for VTG samples.

4. Data Organization, Formats, and Access

Data are organized as a unified training corpus with directory layouts separating images, masks, depth, and annotation JSONs per pointing modality. Key data modalities include:

• Images: RGB (.jpg/.png, max $640\times480$) and optionally depth maps (for 3D RRG; 8-bit normalized from 600–1700 mm).
• Masks: For REG and OFG verification, as binary mask images or bounding box coordinates.
• Instructions: Plain-text natural language, instance-specific.
• Point Labels: Stored as JSON lists (key "points"), with schema varying by modality. REG and OFG contain one 2D coordinate; VTG contains eight.
• Example (REG):

$p\in [0,w]\times [0,h]$1

• Example (VTG):

$p\in [0,w]\times [0,h]$2

No internal validation or test split is reported in the dataset distribution.

5. Statistical Properties and Evaluation Metrics

Embodied-Points-200K provides broad object and spatial coverage:

• Object domains: 70 YCB categories (RRG 3D), 212 objects (HandAL/OFG), ~50,000 web images (REG), hundreds of video sources (VTG).
• Spatial relation types: $\approx$10, including left, right, front, behind, between, center-of, under, over.
• Empirical pointing distances: 20–300 px, varying by scene scale; auxiliary rewards are bounded by $D_{min}=0$ px and $D_{max}=1\,000$ px.

Evaluation draws on human spot-checks (>95% point-in-mask accuracy for REG/OFG), with trajectory feasibility for VTG above 90%. Performance on held-out benchmarks covers both spatial reasoning (CVBench, CRPE, SAT, BLINK, EmbSpatial-Bench) and pointing/trace tasks (RoboRefIt, Where2Place, VABench, Part-Afford, Open6DOR-Position).

6. Role in Training and Benchmarks

Embodied-Points-200K is leveraged in Stage 2 of the reinforced fine-tuning (RFT) curriculum for Embodied-R1, which instills embodied spatial reasoning and guidance capabilities through a mixed, multi-task optimization. Reward design is structured to reflect the specifics of each core task and subtask, including:

• Policy objective:

$$\max_\theta \mathbb{E}_{x\sim D} \mathbb{E}_{y\sim\pi_\theta(\cdot|x)} [ r(y,x) ]$$

• Advantage estimation:

$$\widehat{A}_{i,t} = \frac{r(y_i,x) - \mathrm{mean}_j[r(y_j,x)]}{\mathrm{std}_j[r(y_j,x)]}$$

• Reward components: Format, accuracy, mask inclusion, distance, trace, and simulation success, with task-specific linear combinations (e.g., $p\in [0,w]\times [0,h]$0).

Empirical benchmark results include:

• Zero-shot evaluation:
  • SIMPLEREnv on WidowX arm: 56.2% average success.
  • Real-world xArm-6: 87.5% zero-shot success across 8 tabletop tasks.
  • Improvement over strong baselines: 62% relative increase.

This suggests Embodied-Points-200K is instrumental for robust, generalizable, and policy-guided pointing and manipulation across a spectrum of robotic and simulated environments.

7. Significance and Scope

By unifying referring expression grounding, region and object part pointing, and trajectory generation into a single large-scale “question–verification” corpus, Embodied-Points-200K advances the representation and modeling of spatial interaction for Vision-LLMs in robotics. Its design enables the learning of embodiment-agnostic reasoning—and, through multi-task reinforcement—high-fidelity, generalizable policies for closing the vision-to-action gap in embodied agents. The dataset is a cornerstone of the Embodied-R1 framework and exhibits transfer robustness, supporting progress in both embodied VLM research and practical manipulation policies (Yuan et al., 19 Aug 2025).