Egocentric Referring Video Object Segmentation

• Ego-RVOS is a video segmentation task that isolates active objects in egocentric videos based on natural language action queries.
• It addresses unique challenges such as rapid viewpoint changes, motion blur, occlusions, and dataset biases affecting object-action associations.
• Recent methods like ActionVOS and CERES leverage causal deconfounding and action-aware loss functions to improve segmentation accuracy and suppress false positives.

Egocentric Referring Video Object Segmentation (Ego-RVOS) is a fine-grained video understanding task that aims to segment—at each frame—the object instance actively involved in a specified action, as described by a linguistic query, within first-person (egocentric) video. Distinct from standard third-person or free-form RVOS, Ego-RVOS addresses unique challenges arising from the dynamic and subjective nature of egocentric perception, focusing specifically on the identification of active (rather than merely referenced or present) objects undergoing interaction. This paradigm plays a central role in the automatic understanding of human activities from wearable cameras and facilitates a range of downstream egocentric vision applications.

1. Formal Problem Setting

Ego-RVOS is defined over an egocentric video sequence $\mathcal{V} = \{V_t\}_{t=1}^T$, with frames $V_t \in \mathbb{R}^{H \times W \times 3}$, a natural language action query $\mathcal{A}$ (e.g., "cut apple"), and an (optional) set of candidate object names $\mathcal{O} = \{O_i\}_{i=1}^N$. The output is a temporal sequence of $N$ binary masks $$\mathcal{M}=\{M_t \in \{0,1\}^{N\times H \times W}\}_{t=1}^T$$, where $M_t(O_i)$ specifies the spatial location of object $O_i$ at time $t$ if and only if $O_i$ is actively involved in $\mathcal{A}$. Non-participating ("inactive") objects are suppressed—i.e., their masks must be all-zero.

Contrasted with standard RVOS, which segments any referred object regardless of action state, Ego-RVOS strictly segments, frame by frame, those objects with actual involvement in the described human action (Ouyang et al., 2024, Liu et al., 30 Dec 2025). This distinction underlies both annotation practices and the design of model architectures and loss functions.

2. Key Challenges in the Egocentric Setting

Several factors compound the inherent difficulty of Ego-RVOS relative to third-person RVOS and static-image object segmentation:

• Egocentric viewpoint distortions: Head-mounted cameras induce rapid viewpoint changes, motion blur, and sustained hand-object occlusion.
• Object-action ambiguity and dataset bias: Highly skewed object–action frequency distributions (e.g., "knife–cut" dominating over "knife–wash") produce spurious learning signals, causing models to incorrectly infer involvement based on prior statistics rather than video/language evidence.
• Temporal and appearance variability: Active objects undergo state changes (e.g., breaking, slicing, painting) across the video, rendering static appearance features insufficient.
• No explicit “active” supervision: Most datasets lack direct labels specifying which object instances are active per action (Ouyang et al., 2024).
• Domain shifts: Models pre-trained on third-person or general-domain data struggle to capture egocentric-specific phenomena such as partial visibility and occlusion by the camera wearer’s hands (Liu et al., 30 Dec 2025).

3. Algorithmic Approaches

ActionVOS

ActionVOS (Ouyang et al., 2024) introduces a task variant utilizing action narrations as prompts to focus segmentation on active objects only. The approach builds on ReferFormer as RVOS backbone, with the following additions:

• Classification Head: Per-object feature vectors after the decoder are passed through a linear classifier to predict the probability of “active” involvement, with thresholding at inference to suppress inactive masks.
• Action-aware Labeling Module ($\Phi$): Derives pseudo-labels for object activeness from language mentions, spatial overlap with hand-object masks, and bounding-box intersection.
• Action-guided Focal Loss: Pixel-level focal loss is weighted via empirically set coefficients, modulating the contribution of pixels according to their spatial and semantic relationships to the action prompt, reducing false positives from ambiguous supervision.

CERES: Causal Dual-Modal Intervention

The CERES framework (Liu et al., 30 Dec 2025) offers a general, plug-in approach that wraps around any pre-trained RVOS model. Its architecture is designed to address causal confounding in both language and vision modalities:

• Linguistic Backdoor Deconfounder (LBD): Implements backdoor adjustment by augmenting text features with expected embeddings over the empirical distribution of object–action co-occurrences, mitigating biases from frequent pairings.
• Visual Frontdoor Deconfounder (VFD): Introduces a “mediator” constructed via attention between semantic visual features (RGB) and geometric depth features. This mediation, inspired by Pearl’s front-door adjustment, produces visual representations robust to egocentric confounders, such as motion artifacts and occlusions.
• Temporal Memory (MAttn): Short-term attention over previous frames further stabilizes visual representations, capturing context and mitigating transient noise.

4. Datasets and Annotation Strategies

Current benchmarks for Ego-RVOS include:

Dataset	Description	Notable Annotation
VISOR	Egocentric kitchen actions with dense per-frame masks	VAL set manually relabeled for activeness
VOST	“State-change” events with a single object undergoing transformation	All instances of a state-changing class as positive
VSCOS	Objects with visually evident state changes (e.g., broken eggs)	Pseudo-flagged activeness

No dataset provides explicit, frame-level, gold-standard labels for activeness, necessitating pseudo-labeling modules (e.g., ActionVOS $\Phi$) that aggregate signals from language, hand-object interaction, and spatial overlap (Ouyang et al., 2024).

5. Experimental Protocols and Evaluation Metrics

Typical experimental setups use RVOS backbones such as ReferFormer (with various visual/text encoders), training from checkpoints like ReferYouTube-VOS. AdamW is employed as an optimizer. The main losses include action-guided focal loss (for mask segmentation) and binary cross-entropy (for activeness classification).

Key metrics include:

• p-mIoU / n-mIoU: Mean intersection-over-union (IoU) over positive (active) and negative (inactive) objects.
• p-cIoU / n-cIoU: Cumulative IoU over all positive/negative objects and frames.
• gIoU: Generalized IoU penalizing missed negatives.
• Acc, Precision, Recall, F1: Activeness classification metrics (IoU$>$0.5 as TP).

VISOR validation results demonstrate substantial drops in n-mIoU (inactive objects) from 54.2% (standard RVOS) to 19.0% (ActionVOS with prompts), confirming effective suppression of distracting non-actives while maintaining p-mIoU (active targets) at competitive levels. CERES further improves on ActionVOS across all metrics, particularly on positive and generalized IoUs (Ouyang et al., 2024, Liu et al., 30 Dec 2025).

6. Ablation Analyses and Qualitative Observations

Ablation studies highlight the impact of each architectural and training component:

• Classification Head: Removing activeness thresholding degrades gIoU and activeness classification accuracy.
• Prompt Engineering: “Natural” action formulations (e.g., “knife used in the action of cut apple”) offer better discrimination than noun-only or artificial prompts.
• Loss Function Design: Action-guided focal loss yields improved suppression of false positives.
• Causal Components (CERES): LBD and VFD independently and jointly contribute to improved positive/negative IoU and gIoU. Depth-guided attention provides notable gains, with memory-based context aggregation further enhancing negative suppression.
• Failure Modes: Persistent challenges arise from occluded hands, ambiguous object references, or drastic appearance changes; pseudo-labeling may over-extend active object regions based on proximity rather than real interaction (Ouyang et al., 2024, Liu et al., 30 Dec 2025).

7. Limitations and Directions for Future Research

Identified limitations include reliance on hand-object masks during training (ActionVOS), approximation assumptions in LBD backdoor and mediator expressivity (CERES), short temporal context, and persistent degradation on rare/unseen action–object pairs. Both ActionVOS and CERES acknowledge the need for:

• Reduced dependency on dense pseudo-annotations
• Automated, learned confounder identification and mediation
• Richer relational priors (e.g., scene graphs, affordances)
• Generalization to open-world segmentation without pre-defined object sets
• Extension of causal intervention techniques to broader egocentric video tasks (Ouyang et al., 2024, Liu et al., 30 Dec 2025)

8. Representative Results

Selected quantitative results from VISOR are summarized below (ReferFormer-R101 backbone, validation set):

Method	p-mIoU	n-mIoU	gIoU	Acc
RVOS Upper Bd.	67.7	54.2	43.8	59.1
ActionVOS (w/ Prompt)	65.4	19.0	70.9	82.4
CERES	64.0	15.3	72.4	76.3

On zero-shot transfer (VOST/VSCOS), CERES achieves higher mIoU/cIoU on positive targets than ActionVOS, demonstrating robustness to state changes and novel objects.

Both ActionVOS and CERES have established new state-of-the-art performance in Ego-RVOS, with causal and action-aware mechanisms conferring substantial practical benefits, particularly for the reliable segmentation of truly active objects while suppressing contextually irrelevant distractors (Ouyang et al., 2024, Liu et al., 30 Dec 2025).