EgoDex: Egocentric Dexterity Dataset

• EgoDex is a large-scale, egocentric dataset providing 829 hours of video with high-fidelity 3D hand and upper body pose annotations across diverse tabletop tasks.
• It offers synchronized multimodal streams—including RGB video, 3D skeletons, language annotations, and task metadata—for scalable imitation learning and robotic training.
• Its comprehensive task taxonomy and benchmark results facilitate advanced research in dexterous manipulation and human–robot interaction.

EgoDex is a large-scale dataset explicitly constructed for research in dexterous manipulation, robotics, and egocentric perception. Acquired with Apple Vision Pro headsets worn by uninstrumented humans, EgoDex consists of 829 hours of egocentric video featuring paired, high-fidelity 3D hand and upper body pose annotations across a broad taxonomy of object-centric tabletop tasks. The dataset provides synchronized multimodal streams for imitation learning: video, 3D skeletons, per-joint confidence, language annotations, and task metadata. Its wide coverage, annotation precision, and diversity establish a foundation for scalable learning of hand-object interaction (Hoque et al., 16 May 2025).

1. Data Acquisition and Annotation Pipeline

EgoDex data collection utilized Apple Vision Pro (visionOS 2), enabling high-fidelity, real-time multiview capture without instrumentation or markers on the hands. Key hardware and pipeline characteristics include:

• Capture Modality: Stereo RGB passthrough (1920×1080 @ 30 Hz), wide field of view, real-time on-device SLAM.
• Annotation Output:
  • Full SE(3) pose for 20 upper-body joints (hip to head) and 25 joints per hand, encoded as $4 \times 4$ homogeneous matrices, for each frame.
  • 3D positions of each fingertip.
  • Per-joint visibility/occlusion confidence in $[0, 1]$.
• Processing: At record time, each frame is processed through Apple’s hand-prediction and SLAM networks to extract all joint poses and confidence metrics. Raw data (500 TB) is on-device compressed to 2 TB, enabling efficient torchcodec-based decoding for training.
• Summary Table:

Statistic	Value	Notes
Total video hours	829	≈ 90 million frames (@30 Hz)
Task demonstrations	338,000	Across 194 unique tabletop tasks
Storage footprint	2.0 TB	5×300 GB train, 7 GB test, 97 GB extra
Environment diversity	Dozens of unique locations	Variety of everyday objects

The dataset provides synchronized RGB, multimodal pose, and language annotation streams.

2. Task Taxonomy and Behavioral Diversity

EgoDex defines a comprehensive task taxonomy with 194 distinct tabletop manipulation tasks, systematically curated to cover fine and gross motor operations. The taxonomy comprises:

• Reversible tasks: 76 action-inverse pairs (e.g., “tie/untie shoelaces”, “screw/unscrew”, “open/close cases”), totaling 152 atomic tasks.
• Reset-free tasks: 28 tasks (e.g., “flip a coin”, “playing piano”, “typing”, “page flipping”, “catch/throw a ball”), requiring no environment reset.
• Reset tasks: 14 tasks requiring intervention between episodes (e.g., “basic_pick_place”, “make_sandwich”, “use_chopsticks”, “stack”, “pour”).

Quantitative diversity measures include:

• Over 150 unique verbs, with most verbs exceeding $10^3$ demonstrations—markedly higher density than prior datasets.
• Over 200 distinct object types, clustered by GPT-4 into semantic categories such as “kitchenware”, “tools”, “game pieces”, “electronics”, and “clothing”.
• Average demonstrations per task: $\approx 1,744$.

Behavioral diversity ranges from simple pick-and-place to complex, long-horizon manipulations such as folding laundry, threading, and assembling furniture. While scenes are predominantly tabletops, object and skill diversity is broad.

3. Action Space, State Representation, and Benchmarks

EgoDex establishes a reproducible suite of imitation learning benchmarks for hand trajectory prediction:

• Action Representation: $a_t \in \mathbb{R}^{48}$ composed as follows (per hand: 3D wrist position + 6D wrist orientation + 5 fingers × 3D position = 24 dimensions; total 48).
  • State vector: $s_t =$ all joint poses at $t$.
  • Observations: Egocentric RGB $o_t$. Natural language encoding $l$ via frozen CLIP.
• Benchmark Tasks:
  • Horizon $H$ configurable: $H=30$ (1s), $60$ (2s), $90$ (3s).
• Evaluation Metrics:
  • Best-of-K average positional error (meters, over $H$ timesteps and 12 keypoints: both wrists and ten fingertips):

  $$E_{best-of-K} = \min_{i \in [1..K]} \frac{1}{H \cdot 12} \sum_{t=1}^{H} \sum_{k=1}^{12} \| \hat{g}^{(i,k)}_{t,k} - p_{t,k} \|_2$$

  where $p_{t,k}$ is ground-truth, $\hat{g}^{(i,k)}$ is $i$-th prediction sample. - Avg Distance: mean $E_{best-of-K}$ over all test samples. - Final Distance: average error over 12 keypoints at $t=H$ only.

4. Benchmark Models and Experimental Findings

The paper systematically evaluates model architectures and learning paradigms using the proposed benchmarks:

• Architectures:

  • Decoder-only Transformer (“Dec”)
  • Encoder-decoder Transformer (“EncDec”)
• Learning Algorithms:
  • Behavior Cloning (BC)
  • Denoising Diffusion (DDPM)
  • Flow Matching (FM)
• Inputs: ResNet-encoded images (224×224), CLIP text representations, and kinematic poses.

Main two-second ($H=60$) prediction results ($K=1/5/10$):

Model	Avg Dist (K=1)	Avg Dist (K=10)	Final Dist (K=1)	Final Dist (K=10)
Dec+BC	0.045 m	0.045 m	0.062 m	0.062 m
EncDec+BC	0.044 m	0.044 m	0.060 m	0.060 m
EncDec+FM	0.051 m	0.038 m	0.070 m	0.041 m

Key findings:

• Encoder-decoder marginally outperforms decoder-only models.
• Flow Matching (FM) with EncDec achieves lowest error at $K \geq 5$, up to 34% lower than BC at $K=10$.
• Behavior Cloning is optimal under $K=1$ (single mode).
• Accuracy degrades predictably with increasing $H$: from 0.031/0.049 m (1s), to 0.045/0.062 m (2s), to 0.053/0.069 m (3s) (avg/final).
• Visual goal conditioning ($o_{t+H}$) reduces endpoint error by up to 53%.
• Doubling training episodes correlates with error reduction; no overfitting is observed scaling model size from 200M to 500M parameters.

Qualitative observations highlight highest error on highly dexterous, occluded tasks, with error spikes on fingertip motions during occluded or rapid segments.

5. Applications Across Robotics, Vision, and World Models

EgoDex’s scale, ground truth precision, and diversity unlock several core applications:

• Robotics:
  • Zero-shot transfer to anthropomorphic robot hands, leveraging kinodynamically compatible egocentric policies.
  • Bridging human–robot embodiment gaps via co-training, pretrain–finetune recipes, or representation learning for more sample-efficient robot RL/IL.
• Computer Vision & Perception:
  • Egocentric action anticipation, hand-object interaction mapping, contact/grasp inference, and hand-object segmentation.
  • Multimodal scene understanding by leveraging the joint RGB, 3D pose, and language data.
• Foundation World Models & Video Generation:
  • Training diffusers and autoregressive models to predict future hand–object interactions.
  • Simulating $H$-step egocentric dynamics for planning or RL in both simulated and real environments.

A plausible implication is that EgoDex's structure offers a template for future egocentric datasets targeting both generalizable policy learning and embodied intelligence evaluation.

6. Limitations and Future Directions

EgoDex’s primary scene diversity is limited to tabletop environments. The authors propose the following directions for expansion:

• Increasing visual and environmental diversity via generative background augmentation or out-of-tabletop capture.
• Improving annotation robustness under occlusion or fast dynamics with multi-view setups or enhanced hand-tracking networks.

Collectively, EgoDex establishes a new “ImageNet-scale” foundation for manipulation-centric research across robotics, computer vision, and embodied learning, addressing longstanding data scarcity and annotation barriers in dexterous imitation learning (Hoque et al., 16 May 2025).