RoboCasa Human-50: Benchmark Dataset for Robotics

• RoboCasa Human-50 is a teleoperated demonstration dataset consisting of 1,250 examples across 25 kitchen tasks that serve as fundamental interactions.
• It integrates high-resolution multi-view RGB imaging and proprioceptive feedback for precise 3D perception and temporally extended control.
• The dataset underpins benchmarking with BC-Transformer and geometry-aware models, achieving up to 52.4% success on complex manipulation tasks.

RoboCasa Human-50 is a high-quality, teleoperated demonstration dataset designed for benchmarking imitation learning and generalist manipulation in kitchen-like, everyday robotics environments. It forms a core subset of the larger RoboCasa simulation framework, supplying critical ground-truth data for both algorithm development and comparative evaluation of policy architectures, especially those requiring robust 3D perception and temporally extended control. The dataset includes 1,250 human demonstrations across 25 atomic manipulation tasks, each trajectory captured at high temporal resolution using multi-view egocentric and exocentric observations. Its systematic design, rigorous evaluation protocol, and integration into state-of-the-art learning pipelines have made it a focal point for research on generalist robot policy learning and geometry-aware manipulation (Qian et al., 18 Dec 2025, Nasiriany et al., 2024).

1. Dataset Definition and Composition

RoboCasa Human-50 consists of 25 "atomic" kitchen manipulation tasks, representing fundamental interactions with appliances, furniture, and utensils (e.g., pick-and-place, opening cabinets, pouring, flipping switches). For each atomic task $t$ ($1 \leq t \leq 25$), there are exactly 50 teleoperated human demonstrations, for a total population:

$N = \sum_{t=1}^{25} n_t = 25 \times 50 = 1{,}250$

where $n_t=50$ specifies the demo count per task. This dataset covers a full spectrum of long-horizon and dexterous skills, such as precise insertions, sequential operations, and geometric reasoning in scenes with varied spatial layouts and visual textures (Nasiriany et al., 2024, Qian et al., 18 Dec 2025).

2. Demonstration Structure and Representation

Each demonstration trajectory $\tau$ is a temporally ordered sequence of state-action pairs:

$\tau = (s_1, a_1, \ldots, s_T, a_T)$

with $T$ denoting the variable trajectory length. State $s_t$ at time $t$ comprises:

• Three RGB images: $(I^{\rm hand}_t, I^{\rm left}_t, I^{\rm right}_t)$, including a wrist (egocentric) and two static exocentric cameras (resolution $1 \leq t \leq 25$0).
• Proprioceptive vector $1 \leq t \leq 25$1: 7-DoF end-effector pose ($1 \leq t \leq 25$2) and 6-DoF mobile base pose ($1 \leq t \leq 25$3), for a 13-dimensional state vector.
• For certain variants (e.g., GeoPredict experiments), tracked 3D keypoints for $1 \leq t \leq 25$4 arm joints including the end-effector, aligned with image frames via forward-kinematics.

Action $1 \leq t \leq 25$5 is a continuous vector: 7-dimensional end-effector velocity and 2-dimensional base velocity, for a total of 9 dimensions per timestep. All data is sampled at 25 Hz (timestep 0.04 s), supporting fine-grained temporal credit assignment and high-frequency control (Nasiriany et al., 2024, Qian et al., 18 Dec 2025).

3. Integration in Learning Pipelines

Human-50 demonstrates integration with behavioral cloning (BC) and modern policy architectures. A typical pipeline employs a multi-task BC-Transformer:

• Visual encoder: three ResNet-18 backbones, FiLM fusion, and proprioceptive multilayer perceptron.
• Sequence model: 6-layer Transformer (≈20M parameters) over history length 10, with linguistic goal embedding via CLIP.
• Output: predicts next 10 actions; in closed-loop use, only the first is executed per step, with immediate replanning.
• Loss: mean squared error (MSE) imitation loss

$1 \leq t \leq 25$6

where $1 \leq t \leq 25$7 is the dataset. Training proceeds over 500,000 gradient steps with Adam optimizer (lr = $1 \leq t \leq 25$8, warmup) (Nasiriany et al., 2024).

In geometry-aware approaches such as GeoPredict, Human-50 demonstrations are further leveraged using trajectory-level kinematic prediction, 3D Gaussian geometry forecasting, and masked depth rendering losses. These modules provide additional supervision during training but are omitted during inference for runtime efficiency (Qian et al., 18 Dec 2025).

4. Evaluation Protocols and Metrics

Benchmarking on Human-50 follows a strict evaluation protocol:

• Each model is evaluated on 50 trials per atomic task using five held-out test scenes—distinct in floor plans, textures, and object instances.
• Primary metric is success rate $1 \leq t \leq 25$9:

$N = \sum_{t=1}^{25} n_t = 25 \times 50 = 1{,}250$0

where "success" is task-specific.

• Additional metrics (when applicable) include:
  • Average $N = \sum_{t=1}^{25} n_t = 25 \times 50 = 1{,}250$1 end-effector error

$N = \sum_{t=1}^{25} n_t = 25 \times 50 = 1{,}250$2

• Training losses: trajectory/keypoint MSE, masked depth-rendering $N = \sum_{t=1}^{25} n_t = 25 \times 50 = 1{,}250$3 loss.

Table: Human-50 Success Rates for Representative Methods (Qian et al., 18 Dec 2025)

Method	Success Rate (%)
BC-Transformer	28.8
GWM	39.2
$N = \sum_{t=1}^{25} n_t = 25 \times 50 = 1{,}250$4 (Baseline)	42.3
GeoPredict	52.4

Aggregate success rate for BC-Transformer on Human-50 is 28.8%, with per-skill success ranging from 2–8% for pick-and-place to 42–80% for opening drawers. Composite and long-horizon tasks trained from scratch on 50 demos yield 0–2% success, highlighting the challenge posed by limited demonstration data (Nasiriany et al., 2024, Qian et al., 18 Dec 2025).

5. Comparative Analysis and Scaling Behavior

Empirical analysis reveals that Human-50, while of high quality, is insufficient for robust generalization on its own. Performance plateaus at 28.8% average for atomic tasks and degrades further for composite/long-horizon tasks. Integration with synthetic data sources (e.g., MimicGen-generated datasets with up to 3,000 demos per task) demonstrates a consistent scaling trend in multi-task success (28.8% → 47.6%). Human-50 serves as the essential "seed" for data-driven augmentation, helping mitigate synthetic artifacts such as unnatural motion or unintentional collisions (Nasiriany et al., 2024).

Ablation studies within geometry-aware frameworks demonstrate additive and synergistic effects:

• Incorporating history tracks yields +2.5 pp improvement.
• Explicit future trajectory supervision adds +2.4 pp.
• Predictive 3D Gaussian geometry and depth rendering further improve by +4.1 pp collectively, peaking at 52.4% success when full refinement is used (Qian et al., 18 Dec 2025).

6. Limitations, Trade-Offs, and Directions

Key limitations of Human-50 include restricted scale (1,250 total trajectories) and limited coverage for high-diversity, dexterous skill acquisition. The dataset is most effective as foundational supervision; scaling with high-fidelity synthetic or auto-generated data is required for generalization, especially in tasks demanding novel spatial or visual generalization.

Failure cases include poor handling of extremely thin objects due to coarse voxelization, and sensitivity of predictive kinematic modules to noisy proprioceptive signals. Directions for improvement include multi-resolution 3D geometric supervision, adaptive voxel sizing, incorporation of color/texture in 3DGS, extension to longer predictive horizons, and evaluation on larger RoboCasa variants (e.g., Generated-300) and real-world objects (Qian et al., 18 Dec 2025).

7. Significance and Applications

RoboCasa Human-50 has established itself as a standard for benchmarking imitation learning, vision-language-action policies, and geometry-aware manipulation in simulated domestic environments. It enables rigorous evaluation under held-out, visually diverse scenarios, making it critical for the development and comparative assessment of multi-task and generalist robot policies. Its integration with state-of-the-art policy architectures, from Transformer-based BC to predictive 3D reasoning frameworks, exemplifies its centrality to both algorithmic and systems-level robotics research (Nasiriany et al., 2024, Qian et al., 18 Dec 2025).