HANDAL: Robotic Manipulation & Pose Dataset

• HANDAL is a large-scale, annotated RGB-D dataset that provides precise 6-DoF/9D pose, scale, and affordance labels for handle-bearing objects.
• It comprises 212 object instances across 17 categories with detailed 3D reconstructions, static and dynamic scene captures, employing a semi-automated processing pipeline.
• The dataset underpins robotics research by benchmarking challenges in pose estimation, affordance prediction, and synthetic-to-real domain adaptation.

HANDAL (Household ANnotated Dataset for functionAL robotic grasping and manipulation) is a large-scale, real-world RGB-D dataset specifically constructed to advance research in category-level 6-DoF and 9D object pose estimation, scale estimation, functional affordance detection, and 3D reconstruction for robotic manipulation. HANDAL targets small, manipulable objects with handles—such as pliers, utensils, and screwdrivers—tailored to realistic robotic grasping in hardware and kitchen tool scenarios. It uniquely provides high-fidelity ground truth pose, per-object and per-vertex affordance annotations, and watertight 3D reconstructions acquired through a semi-automated pipeline leveraging only commodity cameras and streamlined processing (Guo et al., 2023, Sommer et al., 27 May 2026, Zhang et al., 13 Oct 2025).

1. Motivation and Scope

HANDAL was designed to address critical limitations of existing vision and robotics datasets. Traditional large-scale 2D benchmarks (COCO, ImageNet) lack manipulable categories and do not provide pose or affordance labels; available 3D/RGBD datasets overwhelmingly focus on large rigid objects (cars, furniture), or offer insufficient instance and category coverage. Precise 3D perception and object function understanding—where and how to grasp a handle—are prerequisites for both robust pick-and-place and functional tool use in robotics. HANDAL systematically focuses on:

• Manipulable, functional object classes of appropriate size and structure for manipulation by robot end-effectors.
• Comprehensive category-level annotation for pose, scale, and functional affordances (here: handles).
• Democratized 3D data collection via off-the-shelf hardware and semi-automated pipelines that avoid crowd-sourcing or specialized apparatus.

2. Dataset Structure and Annotation

HANDAL comprises 212 distinct real-world object instances in 17 handle-bearing categories, divided into toolbox hardware (e.g., claw hammer, pliers, power drill, screwdriver, adjustable wrench, ratchet) and kitchen tools (e.g., spatula, ladle, whisk, measuring cup, pot/pan, strainer, mug). Per category, 12–19 variants differing in size, brand, and material are included for diversity. Data totals 2,253 videos and 308,000 annotated RGB frames, featuring approximately 10 static scene captures per object (under varied backgrounds, lighting, and clutter) as well as 51 dynamic RGB-D sequences where a human manipulates the object for affordance demonstration.

Semi-Automated Processing Pipeline

The HANDAL annotation workflow separates static and dynamic scenes:

• Static Scenes: Captured using a rear HD mobile phone camera. Frame subsampling prioritizes sharpness via Laplacian variance. COLMAP SfM estimates camera poses, while foreground objects are segmented through XMem (1–2 human clicks per video). Neural radiance field reconstruction is performed with Instant-NGP, meshes extracted by Marching Cubes, and scale/orientation corrected using ARKit/ARCore reference poses. All objects are registered to a canonical frame per category (e.g., handle aligned to +x-axis).
• Dynamic Scenes: Acquired via Intel RealSense D415 (RGB-D). XMem and manual segmentation produce masks; BundleSDF reconstructs geometry and tracks 6-DoF pose per frame, which is refined by ICP on depth frames. Canonical registration and pose annotation follow.

For both pipelines, each frame provides R ∈ SO(3), t ∈ ℝ³, and scale s ∈ ℝ (or s ∈ ℝ³ for 9D formulations). Quality checks involve reprojection of the mesh into the original image and frame filtering by manual inspection.

for each video V: F = subsample_frames(V) C = COLMAP(F) M_seg = XMem.segment(F) N = InstantNGP(F * M_seg) mesh_scene = MarchingCubes(N) scale, R_gravity = align_with_AR(C, AR_poses) mesh_scene = apply_transform(mesh_scene, scale, R_gravity) T_init = bbox_alignment(mesh_scene, mesh_ref) T_refined = interactive_refine(T_init) for each frame i: R_i, t_i = extract_pose(C_i, T_refined, scale) if reprojection_error(mesh_ref, R_i, t_i)>threshold: discard frame i

Affordance labels are assigned by manual mesh annotation (vertex/face)—the present focus is on "handle" vs. "non-handle," supporting per-vertex and per-pixel segmentation.

3. Benchmark Tasks and Evaluation Metrics

HANDAL defines challenging benchmarks for the following tasks:

• 6-DoF/9D Object Pose and Scale Estimation: Given an RGB or RGB-D input, predict R ∈ SO(3), t ∈ ℝ³, and s (either uniform or as 3D extents) for objects of a known category but unseen instance. Symmetry-aware evaluation is used for categories (e.g., bottle-like rotational symmetries).
• Affordance Prediction: Predict segmentation of handle regions per frame or per mesh.

Key evaluation metrics include:

• 3D IoU (Intersection-over-Union):

$$\mathrm{IoU}(B,\hat B) = \frac{\mathrm{vol}(B \cap \hat B)}{\mathrm{vol}(B \cup \hat B)}$$

Success is commonly reported as the AUC of IoU at fixed thresholds (e.g., 0.25, 0.5, 0.75).

• ADD/ADD-S: Average Distance (and symmetric variant) over object surface points.
• PCK: Percentage of projected mesh vertices within $\alpha\sqrt{n}$ pixels of ground truth.
• Coarse Rotation Accuracy Acc@θ: Fraction with rotation error $\Delta(R, R^*) < θ$.

Tables below summarize selected results:

Pre-training	Coarse Rotation Acc@30°	3D IoU (%)
ImageNet3D only	2.9 %	–
Every9D-21M only	23.6 %	25.0 %

Reference-free, foundation models trained on Every9D-21M outperform synthetic pre-training by up to an order of magnitude on HANDAL (Sommer et al., 27 May 2026).

4. Methods and Baseline Comparisons

Benchmarking on HANDAL includes both classical and novel architectures:

• Mask R-CNN: On 2D detection and segmentation (COCO pretrain), HANDAL objects achieve BBox AP@[50:95]=0.656, Seg AP@[50:95]=0.562.
• CenterPose: On RGB-only inputs, 3D IoU is 0.340, rising to 0.670 with single-view depth scaling; ADD/ADD-S up to 0.637; PCK up to 0.605.
• Every9D-21M Foundation Model: Single-stage, dense-fusion style, inputting RGB+D and trained with symmetry-aware losses; pre-training on Every9D-21M dramatically boosts HANDAL accuracy.
• Category-Agnostic Transformer (Beyond ‘Templates’): On the dynamic onboarding subset (real-world, novel categories; RGB-D input), achieves AUC IoU25/50/75 of 33.0/10.6/0.2 %, consistently surpassing Any6D and GenPose++ by wide margins (Zhang et al., 13 Oct 2025).

Ablation studies reveal all architectural elements—RGB-D fusion, Mixture-of-Experts inference, and multi-task shape supervision—are required for robust category-agnostic generalization to HANDAL’s challenging object set.

5. Applications and Scientific Impact

HANDAL is expressly constructed to support:

• Development of robust category-level pose estimators for robotics.
• Affordance-aware grasp planning, enabling end-effectors to locate and grasp functional handles.
• 3D synthetic-to-real domain adaptation using canonicalized, watertight meshes.
• Training and evaluation of reference-free, open-world object understanding pipelines for manipulation.

Compared to prior datasets, only HANDAL provides the combinatorial thoroughness of category/instance coverage, full 360° views, dynamic manipulation, accuracy in occlusion and clutter, functional handle labels, and all components sourced from RGB or low-cost RGB-D hardware.

6. Limitations and Prospects

Several practical and technical bottlenecks persist:

• Scene-to-mesh alignment remains semi-manual (∼1 min per scene); robust, fully automatic registration is a key unmet need.
• XMem-based segmentation requires some manual supervision per sequence; advanced, unsupervised, or foundation model-based segmenters would further scale annotation efficiency.
• Meshes lack physically based rendering (PBR) texture quality due to InstantNGP limitations, impeding high-fidelity synthetic data generation.
• COLMAP-based multi-view SfM dominates compute time (∼20 min per scene), suggesting an opportunity for lighter, real-time reconstruction alternatives.

A single researcher generates ∼12 object instances (100 scenes, 15k frames) in roughly 12 hours of manual processing, not including compute (Guo et al., 2023). This suggests the pipeline is tractable but increasingly advantageous as automation improves.

Future directions include extension to additional affordance types, deformable or articulated object annotation, and fully foundation-model driven pipelines for zero-shot semantic and pose annotation.

7. Relationship to Broader Canonicalization and Pose Datasets

HANDAL interacts synergistically with broader-scale real-world datasets, notably Every9D-21M. Pre-training on Every9D-21M, which features 21.8M images and 700 categories with manual canonicalization and symmetry-aware rules, significantly improves HANDAL generalization. The broad, uniform viewpoint coverage, real RGB-D sensor input, semantically meaningful canonical frames, and cross-category symmetry encoding in Every9D-21M mitigate the sim-to-real gap and encourage effective pose learning under challenging occlusions and non-canonical viewpoints. A plausible implication is that HANDAL offers a stringent but real-world-relevant benchmark for new open-set object pose, scale, and affordance models designed for embodied AI and robotic manipulation (Sommer et al., 27 May 2026, Zhang et al., 13 Oct 2025, Guo et al., 2023).