SFHand: Streaming Language-guided 3D Hand Forecasting

Updated 5 July 2026

SFHand is a streaming framework for language-guided 3D hand forecasting that predicts hand type, 2D bounding boxes, 3D pose, and trajectory in real time.
It integrates multimodal encoders, an ROI-enhanced memory layer, and a DETR-style decoder to maintain temporal context and resolve gesture ambiguities using autoregressive predictions.
Paired with the EgoHaFL dataset, SFHand demonstrates significant improvements in trajectory prediction metrics and enables effective transfer to embodied manipulation tasks.

Searching arXiv for the named paper and closely related hand-forecasting work to ground the article in current references. Searching arXiv for "SFHand streaming language-guided 3D hand forecasting". Searching arXiv for the exact title and topic scope. SFHand is a streaming framework for language-guided 3D hand forecasting introduced for real-time egocentric settings in which a system must continually anticipate human hand motion from video and an instruction. It autoregressively predicts a comprehensive next-frame hand state—hand type, 2D bounding box, 3D pose represented by MANO parameters, and 3D trajectory—from a continuous stream of frames and a static natural-language description. The framework combines multimodal encoders, an ROI-enhanced memory layer, and a DETR-style decoder, and is paired with EgoHaFL, a large-scale benchmark with synchronized 3D hand annotations and fine-grained language descriptions (Liu et al., 22 Nov 2025).

1. Scope, task, and problem setting

SFHand addresses streaming autoregressive 3D hand forecasting rather than offline sequence prediction. At time step $t$ , the model receives a textual description $\mathbf{l}$ , the current egocentric frame $\mathbf{v}^t$ , and the current hand state $\mathbf{h}^t$ , and predicts the next hand state $\mathbf{h}^{t+1}$ . During inference, $\mathbf{h}^t$ is the model’s own prediction from step $t-1$ , so multi-step forecasting is obtained by repeated autoregressive rollout (Liu et al., 22 Nov 2025).

The motivation is tied to real-time AR, HCI, assistive robotics, and embodied manipulation. The paper argues that prior 3D hand forecasting methods are ill-suited to these settings because they are largely offline, require accumulated video clips, and are unimodal, so they cannot use language to encode task intent. SFHand is explicitly designed to operate one frame at a time under causal access constraints while retaining temporal context through memory and disambiguating future motion through instruction conditioning (Liu et al., 22 Nov 2025).

The predicted hand state is deliberately broad. It includes hand type (left or right), a 2D hand bounding box, 3D hand pose via MANO parameters, and a 3D trajectory that captures global hand position in space. As a consequence, SFHand is not merely a trajectory extrapolator or a 2D box predictor; it is a structured hand-state forecaster intended to support both perception and control-oriented downstream use (Liu et al., 22 Nov 2025).

2. Model architecture and autoregressive formulation

The model consists of three encoders, an ROI-enhanced memory layer, a DETR-style transformer decoder, and an MLP output head. The encoders are adopted from EgoHOD: a 12-layer GPT-like transformer for text, a CLIP-style architecture for vision, and a lightweight 2-layer transformer for hand state. The hand encoder applies an attention mask to restrict attention to only visible hands. The decoder is a 4-layer transformer driven by learnable detection queries, and training uses Hungarian matching to establish one-to-one correspondences between predicted and ground-truth hand boxes before loss computation (Liu et al., 22 Nov 2025).

The paper gives the core streaming formulation as

$\mathbf{f}_{\text{me}}^t = [\mathbf{f}_\text{l};\; \mathcal{M}(\mathbf{f}_\text{v}^t, \mathbf{f}_\text{h}^t)],$

$\mathbf{h}^{t+1} = \mathcal{D}(\mathbf{q}_{\text{det}}, \mathbf{f}_{\text{me}}^t),$

where $\mathbf{f}_\text{l}$ is the language embedding, $\mathbf{l}$ 0 the visual embedding, $\mathbf{l}$ 1 the hand-state embedding, $\mathbf{l}$ 2 the memory layer, $\mathbf{l}$ 3 the learnable detection queries, and $\mathbf{l}$ 4 the decoder (Liu et al., 22 Nov 2025).

This formulation is significant because it makes the forecasting pipeline explicitly causal. At each step the system has access only to the current frame, the static instruction, the current hand state, and stored historical embeddings; it does not consume future frames. The architecture therefore differs from clip-based forecasters not just in input formatting but in inference semantics. The system is designed to update state continuously rather than to forecast from a preassembled observation window (Liu et al., 22 Nov 2025).

3. ROI-enhanced memory and language conditioning

The main technical novelty is the ROI-enhanced memory layer. At time $\mathbf{l}$ 5, the model concatenates visual and hand embeddings into

$\mathbf{l}$ 6

and stores recent historical embeddings in a fixed-size FIFO queue,

$\mathbf{l}$ 7

The current embedding serves as query, and the queue entries serve as keys and values. A binary ROI mask $\mathbf{l}$ 8, derived from the current 2D hand bounding box, marks visual tokens overlapping the hand region. A learnable scalar $\mathbf{l}$ 9 multiplies this mask and adds a hand-centric bias before softmax, so retrieval emphasizes salient hand regions rather than background context (Liu et al., 22 Nov 2025).

This mechanism matters because the paper shows that vanilla memory is not sufficient and can be harmful. In the memory ablation, no memory yields ADE/FDE of 15.56/18.38, memory without ROI worsens performance to 18.34/22.23, and ROI-enhanced memory improves performance to 12.65/13.08. The authors interpret this as evidence that background tokens can amplify noise and autoregressive error unless historical retrieval is spatially constrained to hand-centric regions (Liu et al., 22 Nov 2025).

Language is incorporated as a static textual instruction tokenized with BPE and encoded by the GPT-like text encoder. The paper presents language as a source of valuable high-level intent and future guidance, particularly for cases where multiple future hand trajectories are visually plausible. This effect appears most strongly in trajectory forecasting rather than local pose estimation. In the input-modality ablation, removing text changes ADE/FDE from 12.65/13.08 to 14.40/15.47, while pose metrics remain similar, which suggests that language primarily improves intent-conditioned motion prediction (Liu et al., 22 Nov 2025).

The combined design implies a specific division of labor: video provides immediate spatial context, autoregressive hand-state input provides dynamical continuity, memory preserves task-relevant historical information, and language resolves semantic ambiguity about future action. The paper’s qualitative example—picking up a bottle, adding dish liquid, and returning the bottle—illustrates this directly: memory allows the model to recover the earlier location of the bottle, while instruction text constrains the longer-horizon action sequence (Liu et al., 22 Nov 2025).

4. EgoHaFL dataset

SFHand is introduced together with EgoHaFL, described as the first large-scale multimodal dataset for language-guided 3D hand forecasting. EgoHaFL is built from a curated subset of Ego4D, augmented with fine-grained sentence-level descriptions from EgoHOD and camera intrinsic annotations from EgoVid. Following EgoHOD’s segmentation strategy, each video is divided into 3-second clips, and each clip becomes an individual training sample (Liu et al., 22 Nov 2025).

For 3D annotation, the paper uses HaMeR to automatically annotate hand states at 16 frames per clip, producing MANO parameters and 3D hand joint positions for all visible hands. Camera intrinsics from EgoVid are then used to convert these annotations into real-world metric trajectories. The dataset reports 247K video clips, 3.95M annotated frames / 3D hand annotations, 242K clips for training, and 5K clips for testing (Liu et al., 22 Nov 2025).

A notable textual detail is that the paper also reports approximately 90 frames per clip in Table 1 while separately stating that HaMeR annotates 16 frames per clip. The text does not explicitly resolve this discrepancy. This suggests a distinction between raw clip duration and the subset of frames selected for 3D hand labeling, but the paper does not formalize that interpretation (Liu et al., 22 Nov 2025).

EgoHaFL is positioned against existing egocentric resources by its conjunction of fine language and 3D hand forecasting annotations at scale. The paper contrasts it with Ego4D, EgoVid, Ego-Exo4D, and EgoHOD, arguing that prior datasets lacked the simultaneous availability of accurate 3D hand information and fine-grained instruction text. In that sense, EgoHaFL is not ancillary to SFHand; it is the dataset infrastructure that makes instruction-conditioned, streaming 3D hand forecasting experimentally tractable (Liu et al., 22 Nov 2025).

5. Training objectives, benchmark results, and ablations

The training objective is a weighted sum over hand type, 2D box, 3D pose, and 3D trajectory: $\mathbf{v}^t$ 0 The individual terms are cross-entropy for hand type, L1 + GIoU for box regression, and L1 for both pose and trajectory. The loss weights are set to $\mathbf{v}^t$ 1 and $\mathbf{v}^t$ 2. Training uses AdamW, an initial learning rate of $\mathbf{v}^t$ 3, 8 GPUs, and batch size 256 per GPU (Liu et al., 22 Nov 2025).

On EgoHaFL, the evaluation metrics are ADE and FDE for trajectory, and JPE and PA-JPE for pose, all in centimeters. The paper compares SFHand to Static, USST, EgoH4, HaMeR, HaMeR + In., and an oracle-style SFHand $\mathbf{v}^t$ 4 variant that uses ground-truth hand state during autoregression (Liu et al., 22 Nov 2025).

Method	Forecasting summary	FPS
EgoH4	ADE/FDE 22.56/22.87; JPE/PA-JPE 5.75/2.30	0.25
HaMeR + In.	ADE/FDE 18.03/17.13; JPE/PA-JPE 3.55/0.93	63.9
SFHand	ADE/FDE 12.65/13.08; JPE/PA-JPE 3.38/0.92	33.4

The paper states that SFHand improves prior work by up to 35.8% in 3D hand forecasting and remains real-time capable at 33.4 FPS. The largest gains are on trajectory prediction, which is consistent with the role of language and memory. Compared with HaMeR + In., SFHand reduces ADE/FDE from 18.03/17.13 to 12.65/13.08 while slightly improving pose quality from 3.55/0.93 to 3.38/0.92 (Liu et al., 22 Nov 2025).

The ablations clarify the contribution of each modality. Without hand input, the model’s ADE/FDE degrades to 19.51/18.47; without text, it degrades to 14.40/15.47; without video, it degrades to 18.10/22.67, with [email protected] dropping to 0.40. The paper notes that removing hand input causes the model to degenerate from autoregressive to regressive, since it no longer conditions on prior predicted hand states (Liu et al., 22 Nov 2025).

6. Transfer to embodied manipulation, limitations, and nomenclature

A secondary contribution of SFHand is the use of its learned representations for embodied manipulation. The transfer protocol freezes the pre-trained encoders and trains a lightweight task-specific prediction head that maps the extracted representations, together with proprioceptive states such as joint velocities, to robot control signals. The paper states that contrastive learning and frame reconstruction objectives are incorporated during pre-training for manipulation transfer, although exact formulas for those auxiliary objectives are not given in the provided text (Liu et al., 22 Nov 2025).

On Franka Kitchen, which contains 5 manipulation tasks and 2 camera viewpoints, SFHand achieves an average success rate of 79.9, exceeding MPI at 78.9, MVP at 76.5, Voltron at 70.5, ImageNet supervised at 39.7, and CLIP at 38.4. On Adroit, which contains 2 dexterous hand manipulation tasks and 3 camera viewpoints, the paper states that SFHand achieves the highest performance and outperforms the previous best model, R3M, by +13.4% average success rate (Liu et al., 22 Nov 2025).

The explicit limitation stated by the paper is that SFHand relies on MANO-based hand representations, which may limit generalization to extreme articulations and severe occlusions. Other constraints are structural rather than framed as formal limitations: the instruction is a static textual description, the benchmark depends on an automatic annotation pipeline centered on HaMeR and camera intrinsics, and inference is autoregressive. A plausible implication is that long-horizon rollout quality remains coupled to the stability of recurrent state prediction and annotation fidelity (Liu et al., 22 Nov 2025).

The term “SFHand” also admits bibliographic ambiguity and benefits from disambiguation. In “Scale Invariant Fully Convolutional Network: Detecting Hands Efficiently”, the formal method name is SIFCN, not SFHand; the data explicitly states that “SFHand” would be only an informal shorthand for that detector rather than a separate framework (Liu et al., 2019). In OpenFS, SF denotes Signing-Hand Focus loss rather than a method named SFHand (Cha et al., 26 Feb 2026). In SHANDS, the relevant name is SHands / Surgical-Hands, a multi-view surgical hand-gesture and error-recognition dataset, again distinct from SFHand (Ma et al., 27 Mar 2026). Within current arXiv usage, SFHand therefore most precisely refers to the streaming framework for language-guided 3D hand forecasting and embodied manipulation introduced in 2025 (Liu et al., 22 Nov 2025).