EgoMask: Egocentric Video Mask Benchmark

Updated 7 July 2026

EgoMask is a benchmark that precisely localizes language-specified objects in first-person videos using pixel-level masks over short to long video spans.
It combines an automatic dual-branch annotation pipeline—leveraging SAM2 and GPT-4o—with human refinement to deliver high-quality segmentation and referring expressions.
The benchmark exposes unique egocentric challenges such as sparse object visibility, small mask areas, and pronounced positional shifts compared to exocentric methods.

EgoMask is a pixel-level benchmark for fine-grained spatiotemporal grounding in egocentric videos, introduced together with EgoMask-Train, a large-scale training dataset constructed by an automatic annotation pipeline. It is formulated in the setting of referring video object segmentation: given an egocentric video and a natural-language expression, the objective is to localize the referred object with segmentation masks across all relevant frames, including short-, medium-, and long-term video spans. The benchmark was designed to address the absence of a pixel-level evaluation resource for long egocentric videos and to expose the gap between exocentric progress and egocentric difficulty in spatiotemporal grounding (Liang et al., 1 Aug 2025).

1. Definition and research scope

EgoMask targets a specific embodied-vision problem: pixel-accurate localization of a language-specified object over time in first-person video. In contrast to box-level grounding or short-clip referring segmentation, the benchmark requires both temporal grounding—identifying when the target appears and disappears—and spatial grounding at mask level in every relevant frame. The intended capability is directly aligned with egocentric AR and robotics scenarios such as highlighting an object used earlier or tracking an item throughout a long daily activity stream (Liang et al., 1 Aug 2025).

The benchmark is accompanied by EgoMask-Train, a training corpus derived from filtered subsets of EgoTracks. EgoMask itself is curated for evaluation and spans short, medium, and long videos, while EgoMask-Train provides scale for adaptation of existing video grounding and RVOS systems. The paper positions this pair as the first pixel-level egocentric spatiotemporal grounding resource and as a vehicle for systematic ego-versus-exo analysis (Liang et al., 1 Aug 2025).

Earlier egocentric resources did not jointly satisfy these requirements. EgoTracks provides dense bounding boxes and category labels for long-term egocentric object tracking but no masks or language; RefEgo provides referring expressions with bounding boxes, but only for short clips and without mask annotations. EgoMask combines pixel-level masks, rich referring expressions, and long-horizon egocentric structure in a single benchmark (Liang et al., 1 Aug 2025).

2. Task formulation and evaluation protocol

The task input is a video $V=\{I_1, I_2, \dots, I_T\}$ and a referring expression $q$ . The output is a sequence of binary masks $\{M_t\}_{t=1}^T$ , where $M_t$ segments the referred object when it is present and is empty when the object is absent. This formulation is stricter than tube-level or box-level grounding because it requires exact shape recovery and correct temporal extent (Liang et al., 1 Aug 2025).

The benchmark distinguishes three frame types. A target frame is a frame in which the ground-truth entity appears. A background frame is a frame in which the target is absent. A predicted frame is a frame in which the model outputs a non-empty mask. Let $T_{\text{gt}}$ denote the set of target-frame indices and $T_{\text{pred}}$ the set of predicted-frame indices. The paper argues that standard region similarity over all frames is misleading in long egocentric videos because background frames dominate the average when the target is sparse (Liang et al., 1 Aug 2025).

To address this, EgoMask reports several complementary metrics. Recall measures temporal grounding performance as

$\text{recall}=\frac{|T_{\text{pred}}\cap T_{\text{gt}}|}{|T_{\text{gt}}|}.$

The metric called all averages per-frame IoU over the full video:

$\text{all}=\frac{1}{T}\sum_{t=1}^{T}\text{IoU}_t.$

A stricter target-frame metric averages IoU only over frames in which the object truly exists:

$\frac{1}{|T_{\text{gt}}|}\sum_{t\in T_{\text{gt}}}\text{IoU}_t.$

A second strict variant averages IoU over all frames where either a ground-truth mask or a predicted mask exists, excluding frames where both are empty. This latter measure penalizes hallucinations on background frames while avoiding inflation from trivially empty segments (Liang et al., 1 Aug 2025).

This evaluation design is central to the benchmark’s contribution. The paper shows that the classic all-frame IoU can behave counterintuitively in egocentric long videos: as recall drops and methods stop predicting on target frames, the score over all frames can increase because many background frames remain trivially correct. The target-focused metrics are therefore presented as more stable indicators of true grounding quality in the egocentric regime (Liang et al., 1 Aug 2025).

3. Dataset construction and annotation pipeline

EgoMask and EgoMask-Train are built from EgoTracks and RefEgo through a two-branch automatic annotation pipeline followed, for the benchmark, by human refinement. The first branch generates masks. For each clip segment, the bounding box in the first frame of each object trajectory is used as a box prompt to SAM2, which produces a mask for that object throughout the clip. A post-processing step then keeps only the mask regions that overlap with the original box annotations, reducing hallucinations and constraining the output to the intended object (Liang et al., 1 Aug 2025).

The second branch generates referring expressions. One route uses direct expression generation: three frames with the clearest views of the object are selected, red boxes are drawn around the target, and GPT-4o is prompted to produce a short expression of at most 10 words and a longer detailed expression. The prompt explicitly discourages references to the red box, image, or frame, and encourages descriptions using visual attributes, spatial relations, and dynamics (Liang et al., 1 Aug 2025).

A second route uses metadata generation followed by templates. GPT-4o is asked to produce an Object Caption, Visual Attributes, and an Affordance phrase. These components are then combined with pre-defined templates to form diverse and unambiguous referring expressions. The reported average lengths are 7.75 words for short expressions, 26.31 for long expressions, 2.98 for captions, 16.19 for visual-attribute descriptions, and 4.72 for affordance phrases (Liang et al., 1 Aug 2025).

For the evaluation benchmark, all automatically generated masks and expressions are refined and verified by human annotators using ISAT with Segment Anything, a semi-automatic annotation interface supporting SAM2. The quality-control protocol evaluates 20% of test annotations using three experts and five-point scores. The average scores are 4.65 for expressions and 4.92 for masks, with error rates below score 3 equal to 2.5% for expressions and 0% for masks. EgoMask-Train, by contrast, retains the automatic outputs directly in order to maintain large scale (Liang et al., 1 Aug 2025).

4. Benchmark composition and egocentric video statistics

EgoMask-Train contains 2,624 videos, 9,592 objects, and 47,968 referring expressions, with average video length 369.94 s. The data are annotated at 1 FPS, whereas EgoTracks was originally sampled at 5 FPS. EgoMask itself contains 315 videos spanning 5 s to 16 min, 700 expressions, and an average expression length of approximately 15 words; all benchmark samples are manually refined and verified (Liang et al., 1 Aug 2025).

Split	Video length	Videos / Objects / Expressions
EgoMask-Short	12.15 s	200 / 200 / 400
EgoMask-Medium	116.30 s	100 / 100 / 200
EgoMask-Long	361.32 s	15 / 50 / 100

The benchmark is explicitly organized to expose egocentric structure across time scales. EgoMask-Short is sourced from RefEgo and has Total Duration 80.31%, Mask Area 1.83%, $\#\text{Traj}=1.66$ , Avg. Traj. Length 57.13%, Disappear Ratio 21.92%, and Adj. Mask IoU 8.51%. EgoMask-Medium consists of segments extracted from long videos and has Total Duration 36.69%, Mask Area 1.87%, $q$ 0, Avg. Traj. Length 11.10%, Disappear Ratio 187.84%, and Adj. Mask IoU 21.15%. EgoMask-Long, drawn from the EgoTracks validation set, has Total Duration 27.48%, Mask Area 1.86%, $q$ 1, Avg. Traj. Length 1.81%, Disappear Ratio 450.29%, and Adj. Mask IoU 19.53% (Liang et al., 1 Aug 2025).

The paper’s most important empirical claim is that egocentric videos differ systematically from exocentric ones along four axes: shorter total duration, sparser continuous trajectories, smaller object size, and larger positional shifts. In EgoMask-Train, objects are visible for only 21.56% of frames, average mask area is 1.20% of the image, average adjacent-frame mask IoU is 14.96%, $q$ 2, Avg. Traj. Length is 1.33%, and Disappear Ratio is 655.82%. For exocentric training sets, the reported corresponding ranges are far less severe: Total Duration 77.51% to 94.33%, Mask Area 5.34% to 10.23%, and Adj. Mask IoU 54.83% to 64.96% (Liang et al., 1 Aug 2025).

Appendix statistics reinforce the same pattern at box level. EgoTracks has video length 369 s, Total Duration 25.23%, BBox Area 2.42%, $q$ 3, Avg. Traj. Length 1.35%, Disappear Ratio 496.31%, and Adj. Bbox IoU 45.07%, whereas exocentric datasets such as MeViS, Ref-DAVIS, and Ref-YT-VOS report much larger object areas, much longer continuous trajectories, much smaller disappearance ratios, and much higher adjacent-frame overlap. These measurements support the benchmark’s claim that egocentric grounding is not merely a domain-shifted version of exocentric RVOS, but a structurally different regime (Liang et al., 1 Aug 2025).

5. Baselines, fine-tuning results, and methodological lessons

The benchmark evaluates three families of models: Grounded-SAM2 as a pipeline baseline using GroundingDINO plus SAM2, Sa2VA as a VideoLLM linked with SAM2, and VideoLISA-3.8B as a VideoLLM linked with image-level SAM rather than SAM2. Grounded-SAM2 uses the highest-confidence detected box in the video as an initial box prompt and then tracks and segments with SAM2. Sa2VA predicts masks for key frames with an LLM and lets SAM2 propagate them through video memory. VideoLISA performs framewise segmentation without video-level propagation (Liang et al., 1 Aug 2025).

The reported numbers show that state-of-the-art methods perform poorly on EgoMask, especially on medium and long videos. On EgoMask-Short, Grounded-SAM2 reaches recall 91.31, all 54.75, target-frame IoU 51.00, and the stricter non-empty-frame IoU 49.95. Sa2VA-26B records recall 70.08, all 48.23, target-frame IoU 39.20, and stricter IoU 37.30. VideoLISA-3.8B attains very high recall, 98.37, but much lower spatial quality, with all 18.14, target-frame IoU 20.94, and stricter IoU 17.85. On EgoMask-Medium, Grounded-SAM2 reports recall 65.85, target-frame IoU 28.23, and stricter IoU 25.73, while on EgoMask-Long it records recall 61.44, target-frame IoU 27.36, and stricter IoU 24.80. Sa2VA and VideoLISA decline more sharply on long videos, with Sa2VA-4B dropping to target-frame IoU 8.68 and stricter IoU 8.11 on EgoMask-Long (Liang et al., 1 Aug 2025).

Fine-tuning on EgoMask-Train improves egocentric performance while preserving, and sometimes improving, exocentric results. Sa2VA-4B (+FT) raises target-frame IoU from 31.01 to 32.92 on EgoMask-Short, from 18.68 to 20.12 on EgoMask-Medium, and from 8.68 to 9.14 on EgoMask-Long. VideoLISA-3.8B (+FT) improves more substantially, from 20.94 to 27.33 on EgoMask-Short, from 11.87 to 17.28 on EgoMask-Medium, and from 12.11 to 14.82 on EgoMask-Long. The paper reports average relative improvements of approximately 41.30% for VideoLISA (+FT) and 5.74% for Sa2VA-4B (+FT) on EgoMask. On exocentric benchmarks, Sa2VA-4B (+FT) improves on MeViS and ReasonVOS and remains essentially unchanged on Ref-DAVIS, while VideoLISA-3.8B (+FT) stays nearly unchanged on Ref-DAVIS and MeViS and improves on ReasonVOS (Liang et al., 1 Aug 2025).

Several methodological lessons follow from the benchmark’s diagnostics. First, SAM2-based video propagation is both more effective and more efficient than framewise SAM-based segmentation in this setting. Grounded-SAM2 runs at 3.17 FPS with highest-confidence initialization and 7.14 FPS in a naive first-box variant; Sa2VA-4B runs at 6.47 FPS; VideoLISA-3.8B runs at 0.42 FPS. Second, initialization is critical. Replacing highest-confidence initialization with a naive first detected box causes marked degradation for Grounded-SAM2, and Sa2VA collapses in cases where the target object does not appear in the first five key frames used to bootstrap SAM2 memory. Third, the factor-wise analyses in the appendix show that performance improves when objects appear more often, are larger, move less, have longer continuous trajectories, and have smaller disappearance ratios. These observations tie model failure modes directly to the benchmark’s four egocentric difficulty axes (Liang et al., 1 Aug 2025).

Within the cited literature, EgoMask is an explicit benchmark name only in the fine-grained egocentric spatiotemporal grounding work. However, adjacent papers use the term or closely related phrasing as an informal shorthand for mask-based egocentric representation learning. A challenge report on masked autoencoders for Ego4D states that a system based on VideoMAE pretraining on Kinetics-400, followed by a shared backbone and task-specific heads for Object State Change Classification and PNR Temporal Localization, is “the kind of setup one might call an ‘EgoMask’ approach.” In that usage, the emphasis is on mask-based self-supervised pretraining and transfer to egocentric downstream tasks rather than on pixel-level grounding (Lei et al., 2022).

A second nearby usage arises in unsupervised domain adaptation for egocentric action recognition. “Adversarially Masked Video Consistency for Unsupervised Domain Adaptation” introduces GADAN and MCL, where a U-Net mask generator learns adversarial masks over egocentric videos to maximize domain discrepancy while the encoder learns domain-invariant features; qualitative visualizations retain hands, held objects, and interaction surfaces while suppressing background clutter. The summary explicitly notes that the paper does not use the exact term “EgoMask,” but that its learned egocentric video masks constitute precisely such a concept in practice (Zhu et al., 2024).

A third related line concerns cross-view object-mask correspondence. In “O-MaMa @ EgoExo4D Correspondence Challenge,” the source object mask in an egocentric view can function as the query mask for retrieving the corresponding destination-view mask. That work reformulates cross-image segmentation as mask matching using DINOv2 features, a Mask-Context Encoder, Ego $q$ 4Exo Cross-Attention, a mask matching contrastive loss, and Hard Negative Adjacent Mining. This suggests a broader ecosystem in which “EgoMask” can denote not only benchmarked referring segmentation, but also the egocentric mask itself as a computational object for cross-view correspondence (Mur-Labadia et al., 6 Jun 2025).

The term has even been extended conceptually to privacy-preserving avatar rendering. “AEGIS: Preserving privacy of 3D Facial Avatars with Adversarial Perturbations” frames its method as a 3D Gaussian-avatar analog of adversarial masking, or “EgoMask in 3D parameter space,” although that is explicitly presented as a conceptual relation rather than official nomenclature. The shared motif is adversarial manipulation of appearance-relevant channels to preserve function while suppressing identity cues (Wolkiewicz et al., 21 Nov 2025).

Taken together, these usages indicate that EgoMask has two layers of meaning in current arXiv-era egocentric vision research. In the strict sense, it denotes the pixel-level benchmark and training set for fine-grained spatiotemporal grounding in egocentric video (Liang et al., 1 Aug 2025). In a broader, informal sense, it functions as a label for egocentric masking paradigms spanning self-supervised video pretraining, adversarial domain adaptation, cross-view mask correspondence, and privacy-oriented perturbation. A plausible implication is that the benchmarked notion of EgoMask may become a focal evaluation target for a wider class of egocentric masking methods as the field converges on long-form, language-conditioned, pixel-level embodied perception.