EgoAnchor: Temporal Grounding in Ego4D Video

• EgoAnchor is an anchor-based temporal grounding framework for Ego4D NLQ that accurately localizes short-duration actions in long egocentric videos.
• It leverages dual visual backbones with high-resolution frame sampling and Transformer-based intra- and inter-modal fusion to capture fine-grained temporal details.
• The method combines anchor classification with boundary regression to improve recall and handle complex long-term temporal reasoning in untrimmed video streams.

EgoAnchor denotes an anchor-based temporal grounding framework for Ego4D natural language query (NLQ), in which an untrimmed ego-centric video $V$ and a language query sentence $Q$ are mapped to the start time $t^s$ and end time $t^e$ of the video segment that best matches the query. The method reformulates egocentric NLQ as an anchor-based detection problem, combining high-resolution frame sampling, dual visual backbones, intra- and inter-modal Transformer fusion, and temporal anchor classification plus boundary regression. It was introduced in the technique report for the Ego4D natural language query challenge, where the central motivation was that Ego4D contains many very short target segments and requires long-term temporal reasoning over videos that are typically longer than eight minutes (Zheng et al., 2022).

1. Problem setting and motivation

The Ego4D NLQ task takes as input an untrimmed ego-centric video $V$ and a query $Q$, and returns the temporal boundaries $(t^s,t^e)$ of the segment matching the query. In contrast to third-person benchmarks such as ActivityNet and Charades-STA, Ego4D videos are captured from a first-person view and contain very fine-grained, short actions embedded in long recordings. Two difficulties are emphasized. First, over $50\%$ of Ego4D queries correspond to segments shorter than $5$ s, so even small temporal misalignment causes a miss. Second, many queries require complex long-term temporal reasoning, including temporal-order questions such as locating an object before it was picked up (Zheng et al., 2022).

This problem framing is significant because it rejects the assumption that methods tuned on third-person temporal grounding transfer directly to ego-centric video. The reported diagnosis is that previous works cannot adapt well to Ego4D mainly because the target segments are excessively short and the temporal dependencies can span minutes of footage. A plausible implication is that temporal resolution and long-range multimodal fusion are first-order architectural constraints rather than peripheral implementation choices.

2. Architecture and anchor mechanism

EgoAnchor casts temporal grounding as a two-stage anchor-based detection pipeline. In the encoding stage, $T$ frames are sampled uniformly from the input video, with the representative setting $Q$0. Visual features are extracted in parallel by a Video Swin Transformer, pre-trained on ImageNet-22K and Kinetics-600, and a CLIP image encoder. The query is tokenized into $Q$1 words and embedded via a pre-trained BERT. Intra-modal fusion is then performed with one Transformer layer on the visual tokens and one Transformer layer on the text tokens. Inter-modal fusion uses $Q$2 cross-modal Transformer layers with hidden dimension $Q$3 and $Q$4 heads, and positional embeddings are added to the visual stream to capture temporal order (Zheng et al., 2022).

In the detection stage, EgoAnchor defines $Q$5 temporal anchors at each of the $Q$6 time steps. Anchor $Q$7 at time $Q$8 covers

$Q$9

where $t^s$0 is the duration of the anchor as a fraction of video length. The anchor set is

$t^s$1

Each anchor $t^s$2 has a center $t^s$3 and a duration $t^s$4. Over the fused multimodal features, two parallel $t^s$5-layer MLP heads are applied to all $t^s$6 anchors: a classification head predicts a confidence $t^s$7 for being a positive match, and a regression head predicts refined boundaries $t^s$8. At inference time, the method selects the top-$t^s$9 anchors by confidence and outputs their refined timestamps.

The anchor mechanism is tailored to the fine temporal granularity of Ego4D. Rather than directly regressing a single interval from a globally pooled representation, the model enumerates dense local hypotheses across the full recording and then refines them. This design is closely aligned with the claim that short-duration queries are especially sensitive to temporal misalignment.

3. Mathematical formulation and optimization

EgoAnchor uses temporal intersection-over-union as the core overlap measure. For anchor $t^e$0 and ground truth $t^e$1,

$t^e$2

The classification target is an IoU-aligned soft label,

$t^e$3

and the classification loss is binary cross-entropy: $t^e$4

Regression is applied only to positive anchors. If $t^e$5 when $t^e$6, then the boundary loss is

$t^e$7

The overall objective is

$t^e$8

where $t^e$9 is the total number of anchors, $V$0 is the number of positive anchors, and $V$1 balances classification and regression. The reported training setting uses a positive-anchor threshold $V$2 and loss weight $V$3 (Zheng et al., 2022).

This formulation is notable for aligning the classification signal directly to overlap quality rather than to a binary anchor label alone. That choice is consistent with the underlying problem regime, where slight temporal displacement can be decisive for short actions.

4. Configuration, training protocol, and reported results

The reported implementation uses $V$4 sampled frames, anchor scales $V$5 of the total video length, and temporal stride $V$6 frame. At each of $V$7 positions, $V$8 anchors are generated, yielding $V$9 proposals. During inference, proposals outside $Q$0 are clipped. Training is performed on the Ego4D NLQ train split for $Q$1 epochs with batch size $Q$2, Adam with learning rate $Q$3 and weight decay $Q$4, and an inverse-square-root learning-rate scheduler. The evaluation metric is $Q$5, defined as the percentage of queries for which at least one of the top-$Q$6 proposals has $Q$7, with $Q$8 and $Q$9 (Zheng et al., 2022).

Study	Setting	Reported values
Test split, single model	Visual = Swin+CLIP, Text = CLIP	IoU $(t^s,t^e)$0: $(t^s,t^e)$1, $(t^s,t^e)$2; IoU $(t^s,t^e)$3: $(t^s,t^e)$4, $(t^s,t^e)$5
Anchor-scale ablation on val	$(t^s,t^e)$6 vs $(t^s,t^e)$7	$(t^s,t^e)$8: $(t^s,t^e)$9 at IoU $50\%$0, $50\%$1 at IoU $50\%$2; $50\%$3: $50\%$4 and $50\%$5
Visual/text features on val	SlowFast+BERT / Swin+CLIP+BERT / Swin+CLIP+CLIP	IoU $50\%$6 $50\%$7: $50\%$8
Sampled frames on val	$50\%$9	IoU $5$0 $5$1: $5$2
Proposal reranking on val	without / with rerank	IoU $5$3: $5$4 $5$5, $5$6 $5$7; IoU $5$8: $5$9 $T$0, $T$1 $T$2

Several points follow directly from these results. The anchor-scale ablation indicates that $T$3 outperforms $T$4 on the validation set, and the frame-sampling ablation shows a large gain from increasing to $T$5 sampled frames. The reported reranking option uses CLIP similarity and an MDETR object-level score, and it improves both $T$6 and $T$7.

5. Interpretation, strengths, and failure modes

The paper’s analysis associates the best anchor scales, $T$8, with the short-duration statistics of Ego4D queries. On an $T$9-minute video, these scales correspond approximately to $Q$00–$Q$01 s, and they are reported to best balance recall against proposal overload. For long-term reasoning, the analysis attributes performance to positional embeddings together with deep cross-modal Transformers, which enable the model to track objects and actions across long videos (Zheng et al., 2022).

Qualitatively, EgoAnchor is reported to correctly localize extremely brief actions, including examples such as picking up a glue bottle, and to answer temporal-order questions. The documented failure mode occurs when multiple similar actions repeat frequently. This failure pattern is consistent with the broader NLQ difficulty in egocentric video: repeated object manipulations can make local visual evidence insufficient unless the model preserves robust long-range temporal context. A plausible implication is that the method’s principal bottleneck is not only proposal density but also identity persistence across repeated, visually similar events.

A common misconception would be to interpret EgoAnchor as merely a direct import of third-person temporal grounding machinery into the ego-centric setting. The reported ablations argue against that reading: higher frame density, short anchor scales, and optional reranking all materially affect outcomes, which suggests that egocentric NLQ imposes a distinct operating regime. The authors also state that they plan to release code, pre-trained backbones for Swin+CLIP fusion, and inference scripts on GitHub.

6. Terminological scope and later reuse of the name

The name “EgoAnchor” has also appeared in later, domain-specific contexts that are not the same method as the Ego4D NLQ detector. In trajectory prediction under ego-view noisy observations, EgoAnchor is a lightweight “intent prior” distillation module inside BiFlow: it computes agent-level and scene-level anchors from a shared latent $Q$02, then modulates decoder features by a FiLM-style transformation,

$Q$03

with reported reductions of minADE and minFDE in ablation under noisy ego-view inputs (Liu et al., 1 Oct 2025). In collaborative augmented reality, the underlying idea of “anchors” refers to repeatedly detected facial landmarks or rigidly mounted glasses points that condition ego-motion alignment between two independent VIO coordinate systems, using a closed-form minimal solver for a Quadratic Eigenvalue Problem and Gaussian Belief Propagation refinement (Micusik et al., 2020). In a separate summary of multi-agent LLM deliberation, “anchor” denotes a hidden internal belief $Q$04 that biases each agent’s opinion dynamics through a term of the form $Q$05, allowing trajectories to escape the convex hull of initial beliefs (Pokharel et al., 17 Jun 2026).

This suggests that “EgoAnchor” is best understood as overloaded terminology rather than as a single trans-domain formalism. In the strict historical sense established by the 2022 Ego4D challenge report, however, EgoAnchor refers to the anchor-based temporal grounding framework for natural language query in egocentric video (Zheng et al., 2022).