EgoLoc: 3D Localization for Egocentric Vision

• EgoLoc is a modular pipeline for 3D localization in egocentric videos that leverages SfM, 2D visual-query retrieval, and confidence-weighted multiview fusion to estimate object displacements accurately.
• The system addresses camera pose recovery as a critical bottleneck, boosting test-set success from 8.71% to 87.12% overall by replacing traditional relocalization with COLMAP-based SfM.
• Its design decouples 2D object detection from 3D geometric reasoning, setting a benchmark in the Ego4D VQ3D task and inspiring hybrid extensions like EgoLoc-v1 for improved pose coverage.

EgoLoc is a modular egocentric-vision pipeline introduced for Visual Queries with 3D Localization (VQ3D) in the Ego4D Episodic Memory benchmark. In its canonical 2022 form, it takes an egocentric video clip, a query object crop, and a query frame, and estimates the object’s 3D location relative to that query frame by coupling egocentric Structure-from-Motion (SfM), 2D visual-query retrieval, monocular depth, and confidence-weighted multiview fusion (Mai et al., 2022). Its central diagnosis is that prior VQ3D systems were bottlenecked less by the abstract idea of 2D-to-3D lifting than by the low number of valid camera poses, quantified as QwP (“Queries with Poses”), and its reported test-set result of 87.12% overall success rate made that pose bottleneck explicit at benchmark scale (Mai et al., 2022).

1. Benchmark setting and task definition

EgoLoc was proposed for the Ego4D VQ3D task. Given an egocentric video $\mathcal{V}$, a query object $o$ defined by a single visual crop $v$, and a query frame $q$, the objective is to estimate the relative displacement vector

$\Delta d = (\Delta x, \Delta y, \Delta z)$

that specifies the 3D location where the query object was last seen, with respect to the reference system defined by the 3D pose of the query frame (Mai et al., 2022).

This formulation places EgoLoc at the junction of two benchmark subproblems. VQ2D localizes the queried object in 2D video frames, while VQ3D lifts those 2D localizations into 3D using camera geometry and depth. EgoLoc’s contribution is therefore not a replacement of VQ2D, but a reorganization of the interface between 2D retrieval and 3D geometric reasoning. The method treats camera pose availability as a first-order variable because VQ3D fails whenever the query frame or the response frames lack valid poses. In benchmark terms, this is reflected by QwP, which effectively upper-bounds end-to-end success (Mai et al., 2022).

The benchmark reports Success, Success$^*$, L2, Angle, and QwP. EgoLoc emphasizes that QwP is structurally decisive: if the system cannot recover the relevant poses, it cannot emit a valid 3D displacement. This reframing distinguishes EgoLoc from earlier baseline formulations that treated camera relocalization as an upstream utility rather than the dominant systems bottleneck (Mai et al., 2022).

2. Modular pipeline and geometric formulation

EgoLoc is explicitly a modular, multi-stage pipeline, not an end-to-end learned model. Its stages are: camera intrinsic and pose estimation from egocentric video using COLMAP SfM; 2D object retrieval with a Siamese visual-query detector; peak-response selection; monocular depth estimation; backprojection of multiple 2D responses into 3D; confidence-weighted aggregation of the resulting 3D hypotheses; and transformation of the fused 3D estimate into the query-camera coordinate frame (Mai et al., 2022).

The core lifting equation is

$$[x_{p_i},y_{p_i},z_{p_i},1]^T = T_{p_i}\, d_{p_i} K^{-1}[u_{p_i},v_{p_i},1]^T,$$

where $T_{p_i}$ is the response-frame pose, $K$ is the intrinsic matrix, $(u_{p_i},v_{p_i})$ is the centroid of the selected 2D box, and $o$0 is the estimated depth at that centroid (Mai et al., 2022). EgoLoc first predicts an object center in world coordinates and only afterward converts it into the query-frame coordinate system: $o$1 This decomposition is important because it makes multiview fusion natural: multiple response frames can vote on a single world-space object hypothesis before final coordinate conversion (Mai et al., 2022).

The pipeline’s multiview stage is confidence-aware. EgoLoc defines an aggregation operator

$o$2

and uses a confidence-weighted average based on the 2D detector score. Conceptually, the detector’s confidence is not discarded after frame selection; it is reused as a weight in 3D fusion. This is one of the paper’s main structural departures from earlier “last detection then unproject” designs (Mai et al., 2022).

3. Camera pose estimation as the dominant bottleneck

EgoLoc’s first major intervention is to replace scan-based frame relocalization with egocentric SfM using COLMAP directly on the video. The earlier Ego4D baseline localized frames by matching real egocentric video to rendered Matterport scan views, which EgoLoc characterizes as brittle because of a scan-to-video domain gap involving illumination, appearance, incompleteness, blur, fisheye distortion, and atypical egocentric viewpoints (Mai et al., 2022).

The SfM branch uses sequential matching over temporally adjacent frames, with implementation choices including selection of 100 contiguous non-blurry frames using variance of Laplacian $o$3 to initialize intrinsics, a RADIAL_FISHEYE camera model, and a sequential matcher with window_size = 10 (Mai et al., 2022). Because monocular COLMAP reconstruction has unknown global scale and its own coordinate frame, EgoLoc subsequently performs a Sim3 alignment into the Matterport scan frame using at least three rendered scan images with known camera poses (Mai et al., 2022).

This design significantly increases pose coverage. The paper’s strongest systems-level result is that replacing only the baseline camera-pose component with EgoLoc’s SfM front end boosted test-set success from 8.71% to 77.27%, with QwP rising from 15.15% to 90.15%; the remaining EgoLoc modules then raised overall success further to 87.12% with QwP 90.53% (Mai et al., 2022). This sharply isolates pose recovery as the task’s central hidden variable.

The method remains constrained by SfM failure modes endemic to egocentric video: motion blur, rapid head rotations, dynamic objects and hands, fisheye distortion, unusual viewpoints, and short translational baselines. EgoLoc does not claim that SfM solves those issues universally; rather, it demonstrates that direct egocentric reconstruction is much more effective than the earlier scan-render relocalization front end in this benchmark setting (Mai et al., 2022).

4. 2D query matching, peak selection, and multiview fusion

EgoLoc revises the VQ2D stage in a way that is tailored to downstream geometry. For each frame, a pretrained Faster R-CNN-style RPN with FPN backbone proposes regions, RoI-Align extracts proposal features, and a Siamese head compares those proposal features to the query crop. The similarity score is

$o$4

with $o$5 the detector backbone feature extractor and $o$6 a projection module (Mai et al., 2022).

Instead of inheriting a tracker and trusting its final response, EgoLoc smooths the per-frame score sequence with a median filter of kernel size 5 and selects local peaks using SciPy find_peaks constraints such as distance = 25, width = 3, prominence = 0.2, wlen = 50, and rel_height = 0.5 (Mai et al., 2022). This tracker removal is not incidental. The paper argues that tracking drift and blurry tracklets degrade 3D lifting, whereas peak detections provide cleaner object centers for depth extraction and backprojection.

Depth is estimated with DPT at the bounding box centroid, and each posed peak is lifted into world space through the backprojection equation. EgoLoc then aggregates multiple posed peaks. On validation, moving from a single last detection to multiple detection peaks improves success from 78.05% to 80.49%, and among multiview fusion operators, the Det Weighted variant performs best (Mai et al., 2022).

The paper also studies triangulation and reports that it is much worse than DPT-based monocular lifting in this setting. On validation, triangulation from detection peaks reaches 56.1%, triangulation from ground-truth track reaches 56.71%, while DPT-based EgoLoc reaches 80.49% (Mai et al., 2022). The stated reasons are egocentric short baselines, sensitivity to pose precision, and undistortion quality. A common misconception is therefore that “more geometry” should imply classical triangulation; EgoLoc’s empirical result is the opposite in Ego4D VQ3D.

5. Empirical performance, extensions, and direct descendants

EgoLoc reports 87.12% overall success rate, 96.14% Success$o$7, 1.86 L2, 0.92 Angle, and 90.53% QwP on the Ego4D VQ3D test set, establishing the reported state of the art for that benchmark instance (Mai et al., 2022). The paper’s most revealing ablation is the pose-only replacement discussed above, because it shows that the majority of the baseline-to-EgoLoc gain originates in camera-pose recovery rather than only in 2D retrieval refinements.

A direct extension, EgoLoc-v1, retains the EgoLoc pipeline after pose estimation but changes the front end from pure SfM to a hybrid SfM + camera relocalization pipeline. It runs SfM on egocentric video, performs 2D–3D matching between existing 3D scans and 2D video frames, estimates additional poses via PnP, and takes the union of poses from SfM and PnP (Mai et al., 2024). The motivation is explicit: EgoLoc still fundamentally relies on SfM over egocentric video, and COLMAP/SfM does not always recover poses for all frames; the hybrid design is intended to increase pose coverage.

On the Eval AI test-set leaderboard, EgoLoc-v1 reports

$o$8

surpassing EgoLoc by 1.5% in the paper’s emphasized metric, overall success rate (Mai et al., 2024). The gain is modest, and the authors explicitly interpret that modesty as evidence that matching between Matterport scans and egocentric videos remains difficult.

Within the broader Ego4D VQ3D lineage, EgoCOL is a closely related camera-pose-centric system that uses sparse SfM reconstructions and a weighted 7-DoF Procrustes mapping to the scan coordinate system, obtaining much higher frame pose recovery than the Ego4D baseline and ranking second in the CVPR 2023 challenge (Forigua et al., 2023). EgoLoc and EgoCOL share the premise that camera-pose coverage is decisive, but EgoLoc’s distinctive synthesis is the combination of egocentric SfM, detector-centered peak selection, and confidence-weighted multiview 3D object fusion.

6. Limitations, broader context, and name reuse

EgoLoc is not a generic egocentric localization system in the SLAM sense. Its output is not a full camera trajectory for its own sake, but a queried object’s displacement vector relative to a designated query frame. Even so, the method exposes several limitations that remain central to egocentric 3D reasoning. The most persistent is camera pose failure in difficult first-person video. Another is the assumption of short-term geometric consistency across views: the paper gives an illustrative failure when the queried object is being moved by the wearer, because confidence-weighted multiview fusion implicitly prefers a static or near-static object hypothesis (Mai et al., 2022).

There is also a benchmark-structural issue. EgoLoc’s 2022 formulation no longer depends on scans for front-end camera localization, but Matterport scans remain necessary for alignment and evaluation. A plausible implication is that the method is more benchmark-native than deployment-agnostic: it solves a real systems bottleneck while still inheriting the benchmark’s scan-based coordinate convention.

Finally, the label “EgoLoc” is no longer unique within arXiv literature. In 2025, the same name was reused for “EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videos,” a zero-shot method for localizing contact and separation timestamps in hand-object interaction, rather than 3D object position in VQ3D (Ma et al., 17 Aug 2025). This reuse does not alter the technical identity of the 2022 VQ3D pipeline, but it does mean that “EgoLoc” has become a shared method name across distinct egocentric-video subfields.