EgoHaFL: Egocentric 3D Hand Forecasting Dataset

• EgoHaFL is a large-scale egocentric dataset that unifies video, fine-grained language instructions, and precise 3D hand annotations for streaming forecasting.
• It comprises 247K three-second video clips with 3.95M annotated frames, offering synchronized multimodal data tailored for online prediction tasks.
• The dataset underpins real-time SFHand models, achieving state-of-the-art forecasting improvements and facilitating transfer to embodied robotic manipulation.

EgoHaFL is the Egocentric 3D Hand Forecasting dataset with Language instruction, introduced as the dataset foundation of "SFHand: A Streaming Framework for Language-guided 3D Hand Forecasting and Embodied Manipulation" (Liu et al., 22 Nov 2025). It is described as the first large-scale dataset to provide synchronized, detailed natural language descriptions alongside precise 3D hand pose and trajectory annotations for egocentric videos, and it was created to enable instruction-aware, streaming 3D hand forecasting. In the paper’s formulation, EgoHaFL supplies the synchronized multimodal supervision required for real-time, online prediction from continuous egocentric video plus natural-language task instructions, while also supporting transfer of learned hand-motion representations to embodied manipulation (Liu et al., 22 Nov 2025).

1. Origin and motivating problem

EgoHaFL was introduced because prior hand forecasting datasets and benchmarks were not well matched to real-time, online prediction from continuous egocentric video plus natural-language task instructions (Liu et al., 22 Nov 2025). The paper identifies three simultaneous requirements for this setting: streaming video input rather than offline access to an accumulated clip, language guidance that communicates task intent, and accurate 3D hand annotations over time. Without a dataset containing all three, the paper argues that a model such as SFHand cannot be properly trained or evaluated.

The dataset therefore addresses a specific bottleneck in multimodal egocentric forecasting. Existing resources were characterized as lacking reliable 3D hand annotations, providing only coarse text, or not being organized as short forecasting-friendly clips. EgoHaFL was designed to close that gap by providing synchronized supervision for predicting future hand state in a streaming setting, rather than for offline sequence processing or coarse action recognition alone (Liu et al., 22 Nov 2025).

This role is central to the paper’s broader contribution. EgoHaFL is not presented as an isolated data release; it serves as the training and evaluation substrate for a streaming autoregressive model that predicts future hand state from video, language, and current hand state. In that sense, the dataset and the forecasting formulation are tightly coupled.

2. Scale, structure, and annotation schema

EgoHaFL is explicitly described as large scale (Liu et al., 22 Nov 2025).

Item	Value
Video clips	247K video clips
Annotated frames / 3D hand annotations	3.95M annotated frames / 3D hand annotations
Frames per clip	about 90 frames per clip
Training split	242K clips
Test split	5K clips
Clip duration	3-second clips

The dataset is built from a curated high-quality subset of Ego4D and leverages the fine-grained, sentence-level language descriptions from EgoHOD together with camera intrinsic annotations from EgoVid (Liu et al., 22 Nov 2025). Its pre-segmentation into 3-second clips is important because it makes the samples naturally fit a forecasting setup.

The annotation schema is multimodal and synchronized. EgoHaFL contains fine-grained language instructions / descriptions, 3D hand pose, 3D hand trajectory, 2D hand bounding boxes, hand type with labels left / right / background, MANO parameters, and 3D hand joint positions (Liu et al., 22 Nov 2025). The paper further describes the hand state $\mathbf{h}$ used by SFHand as containing the hand type, 2D bounding box, 3D pose, and 3D trajectory; in the method description, the 3D pose is represented by MANO parameters and the trajectory is represented in metric space.

Because these modalities are synchronized frame-by-frame, EgoHaFL supports multimodal forecasting rather than post hoc annotation alignment. This synchronization is the key dataset property that makes language-conditioned streaming prediction operational at scale.

3. Construction pipeline and metric reconstruction

The paper describes a mostly automated pipeline assembled from existing data sources and annotation tools rather than hand-labeling every frame from scratch (Liu et al., 22 Nov 2025). The workflow begins from a curated subset of Ego4D videos, adopts fine-grained sentence-level descriptions from EgoHOD, and uses EgoVid camera intrinsics to support metric 3D trajectory reconstruction. Each video is then segmented into multiple 3-second clips, following EgoHOD’s segmentation strategy.

For hand annotation, the pipeline applies HaMeR to automatically annotate 3D hand poses at 16 frames per clip. It then produces MANO parameters and 3D hand joint positions for all visible hands. The paper states that these 3D hand annotations are converted into real-world metric coordinates using the camera intrinsics, yielding physically meaningful trajectories (Liu et al., 22 Nov 2025).

This construction strategy matters for scale. The dataset is described as being bootstrapped from strong existing models and aligned metadata and then organized into forecasting samples, rather than collected through exhaustive manual labeling. A plausible implication is that the design prioritizes forecasting-oriented structure and synchronized modalities over a purely annotation-from-scratch protocol, but the paper’s explicit claim is that the pipeline makes the dataset scalable.

4. Role in the streaming forecasting formulation

EgoHaFL is the dataset that provides the synchronized training data needed for the paper’s streaming multimodal formulation (Liu et al., 22 Nov 2025). In that setup, at each time step $t$, the model receives a language instruction $\mathbf{l}$, a streaming video frame $\mathbf{v}^t$, and the current hand state $\mathbf{h}^t$, and predicts $\mathbf{h}^{t+1}$. The hand state is defined as a comprehensive representation containing hand type, 2D bounding box, 3D pose, and 3D trajectory.

SFHand combines a streaming autoregressive architecture with an ROI-enhanced memory layer and a DETR-style decoder (Liu et al., 22 Nov 2025). EgoHaFL’s synchronized 2D bounding boxes are not incidental to that architecture: the ROI-enhanced memory mechanism uses the 2D bounding box from $\mathbf{h}^t$ to construct an ROI mask over visual tokens. During training, the mask uses the ground-truth box; during inference, it uses the model’s predicted box.

The value of EgoHaFL is therefore not only that it contains 3D hand labels and text, but that its annotations are structured to support online autoregressive prediction. The paper emphasizes that older forecasting methods are typically offline, require accumulated sequences, and are therefore unsuitable for low-latency settings such as AR or robotic assistance. EgoHaFL’s clip structure and synchronized supervision make it natural to train and test such streaming prediction one frame at a time (Liu et al., 22 Nov 2025).

5. Relationship to prior egocentric datasets

The paper contrasts EgoHaFL with several prior egocentric resources and frames its novelty around the combination of fine-grained language + accurate 3D hand annotations together, pre-segmented short clips, much denser frame-level annotations than prior multimodal datasets, and explicit suitability for forecasting (Liu et al., 22 Nov 2025).

Ego4D is described as having only coarse text and unreliable 3D hand annotations for learning. EgoVid provides very large video scale, but only coarse labels and not dense fine-grained hand forecasting supervision. Ego-Exo4D has 3D hand annotations, but the text is coarse. EgoHOD has fine text, but not the same scale or forecasting-oriented 3D hand annotation setup (Liu et al., 22 Nov 2025).

In the paper’s comparison table, EgoHaFL is summarized as having fine text and 3.95M 3D hand annotations across 247K videos/clips, whereas the compared datasets do not provide the same combination of fine language and forecasting-oriented 3D hand supervision. This distinction is important because the target task is not generic egocentric understanding; it is instruction-aware forecasting under streaming constraints.

A common misconception is to treat EgoHaFL as merely another Ego4D-derived annotation layer. The paper’s description is narrower and more specific: EgoHaFL is organized around the requirements of streaming 3D hand forecasting, with short clips, synchronized multimodal supervision, and metric hand trajectories that support an autoregressive next-state formulation.

6. Benchmarks, metrics, and empirical role

EgoHaFL is used in two main ways in the paper: as the primary benchmark for 3D hand forecasting and as the pretraining source for downstream embodied manipulation transfer (Liu et al., 22 Nov 2025).

For the primary benchmark, forecasting on the EgoHaFL test split is evaluated with ADE (Average Displacement Error), FDE (Final Displacement Error), JPE (Joint Position Error), and PA-JPE (Procrustes-Aligned Joint Position Error). The paper states that trajectory error is measured in centimeters with ADE and FDE, and pose error is measured with JPE and PA-JPE after rigid Procrustes alignment. On EgoHaFL, SFHand achieves new state of the art, with improvements of up to 35.8% over prior work, and reports ADE 12.65, FDE 13.08, JPE 3.38, PA-JPE 0.92, and 33.4 FPS (Liu et al., 22 Nov 2025). The oracle-input variant SFHand$^*$ is reported as achieving ADE 10.39, FDE 9.74, JPE 2.91, PA-JPE 0.79, and 58.8 FPS.

The paper also reports that language, video, and past hand state all help, and that the ROI-enhanced memory layer improves performance, especially for trajectory prediction. Notably, a naive memory without ROI can be worse than no memory, whereas the full ROI-enhanced version is best (Liu et al., 22 Nov 2025). Within the article’s scope, that observation matters because EgoHaFL provides the synchronized annotations—particularly boxes, pose, and trajectory—on which those ablations depend.

For downstream evaluation, the pretrained representation is transferred to Franka Kitchen and Adroit. These are not EgoHaFL itself, but the dataset is used to pretrain SFHand so that the learned representations can improve manipulation policies. The paper reports 79.9% average success rate on Franka Kitchen and an improvement of +13.4% average success rate over the previous best model on Adroit (Liu et al., 22 Nov 2025). The stated interpretation is that forecasting human hand motion in a streaming, language-guided way learns physically meaningful representations useful for robot control.

7. Position within adjacent egocentric motion research

EgoHaFL occupies a specific niche within egocentric motion research: it is a dataset for streaming, language-guided 3D hand forecasting, rather than a general full-body motion-estimation system or an object-trajectory generation benchmark. This distinction becomes clearer when it is compared with neighboring work.

"EgoAllo: Estimating Body and Hand Motion in an Ego-sensed World" addresses recovery of the wearer’s human motion in the allocentric frame of the world from egocentric SLAM poses and images, and it uses estimated body pose as a kinematic prior for hand estimation (Yi et al., 2024). Its task is human motion estimation from a head-mounted device, with outputs including SMPL-H body joint rotations, body shape or height, and hand-related parameters. That setting differs from EgoHaFL’s role as a dataset for predicting the next hand state from streaming video and language.

"EgoFlow: Gradient-Guided Flow Matching for Egocentric 6DoF Object Motion Generation" is likewise related in spirit but not identical in task definition (Saroha et al., 1 Apr 2026). EgoFlow is a scene-conditioned generative model for egocentric 6DoF object trajectory synthesis, conditioned on multimodal egocentric observations and scene context. The paper explicitly characterizes it as connected to EgoHaFL-style problems because both concern egocentric, physically grounded motion generation, while also stating that EgoFlow’s concrete task is object pose trajectory generation rather than hand trajectory forecasting.

Placed against these neighboring directions, EgoHaFL is best understood as a dataset intervention targeted at a missing problem formulation: instruction-aware, online, hand-centric forecasting with synchronized video, language, and metric 3D supervision. Its significance lies less in proposing a new body model or generative prior than in supplying the multimodal substrate required for that forecasting regime and in demonstrating that such supervision can transfer to embodied manipulation (Liu et al., 22 Nov 2025).