SEER-VAR: Egocentric AR & Semantic SLAM

• SEER-VAR is a framework for vehicle AR that decouples in-cabin and road scenes using depth-guided semantic segmentation and dual context-aware SLAM branches.
• It integrates vision–language grounding with mask-based object proposals to accurately differentiate and track dynamic environments.
• The system employs LLM-based overlay recommendation and is validated on the EgoSLAM-Drive dataset, demonstrating high spatial consistency and perceptual fidelity.

SEER-VAR (Semantic Egocentric Environment Reasoner for Vehicle Augmented Reality) is a framework advancing egocentric vehicle-based augmented reality (AR) by synergistically combining semantic scene decomposition, depth-guided vision–language grounding, dual context-aware SLAM branches (CASB), and LLM–based AR overlay recommendation. SEER-VAR is designed to address the challenge of robust, contextually anchored AR in dynamic, mixed in-cabin and out-of-cabin driving environments, and is supported by the EgoSLAM-Drive dataset for systematic benchmarking (Lai et al., 24 Aug 2025).

1. Problem Definition and Design Philosophy

SEER-VAR formulates vehicle AR as a joint semantic understanding and localization task in highly dynamic egocentric video streams. The input consists of a monocular RGB video stream $$\mathcal{I}(t_i) \in \mathbb{R}^{3 \times H \times W}$$ (1408×1408 at 30 FPS), capturing both in-cabin (“intra”) and external road (“extra”) scenes, and vehicle telemetry $\mathcal{X}(t_i)$ (speed, fuel, etc.).

The system decouples the egocentric video into two semantic contexts:

• Intra: The physical coordinate frame fixed to the vehicle cabin.
• Extra: The coordinate frame aligned with the outside world.

Rigid-body transforms $${}^{in}\!T_{C_{t_i}}, {}^{ext}\!T_{C_{t_i}} \in SE(3)$$ map camera points to these respective contexts.

The motivation for semantic decomposition is rooted in pronounced differences between cabin and road scene dynamics, depth ranges, and geometric structures. Standard monocular SLAM approaches often fail to initialize or drift in such mixed settings due to conflicting depth cues and pervasive motion. Decoupling enables reliable context-specific tracking and coherent AR overlay fusion (Lai et al., 24 Aug 2025).

2. Depth-Guided Vision–Language Grounding

SEER-VAR leverages a hybrid vision pipeline to effectuate robust semantic separation. The pipeline synthesizes depth estimation, vision–language object grounding, and mask-based context extraction as follows:

• Depth Estimation: Each frame acquires dense depth via Depth Anything V2, $$D(t_i) = d_{dep}(\mathcal{I}(t_i)) \in \mathbb{R}^{H \times W}$$.
• Object Proposals and Segmentation: Grounding DINO generates vision–language object proposals using prompts (e.g., “steering wheel, person, car”), and SAM 2 is used for segmentation and mask-based tracking to produce binary masks $\mathcal{B}(t_i)$.
• Histogram-Based Thresholding: The system computes a 256-bin histogram $H(k)$ of the depth map, identifying distinct peaks $k_1, k_2$ and minimum $k_{min}$ to partition near (cabin) and far (road) regions.
• Context Mask Extraction: In-cabin and out-of-cabin masks are generated as

$${}^{in}\!\mathcal{V}^\alpha = \mathbb{1} \left( (D \odot (1-\mathcal{B})) < k_{min} \right),\quad {}^{ext}\!\mathcal{V}^\alpha = \mathbb{1} \left( (D \odot (1-\mathcal{B})) > k_{min} \right)$$

yielding RGBA-masked intra/extra views for downstream tasks.

An optional loss term for grounding ($\mathcal{L}_{ground}$) may be used when fine-tuning, based on binary cross-entropy between pseudo-ground-truth and predicted masks.

3. Dual Context-Aware SLAM Branches (CASB)

SEER-VAR instantiates two parallel ORB-SLAM3 pipelines—one for each semantic context—operating on features exclusively from their respective masked regions:

• Feature Extraction: SuperGlue extracts matches within $\mathcal{X}(t_i)$0 and $\mathcal{X}(t_i)$1.
• Bundle Adjustment: For each branch $\mathcal{X}(t_i)$2, SEER-VAR minimizes reprojection error:

$\mathcal{X}(t_i)$3

and $\mathcal{X}(t_i)$4 denotes 3D map-points.

• Pose Optimization: Gauss–Newton updates are performed per iteration:

$\mathcal{X}(t_i)$5

with $\mathcal{X}(t_i)$6 as the reprojection residual and $\mathcal{X}(t_i)$7 its Jacobian.

• Loop Closure Detection: Each branch runs covisibility and place-recognition independently, with boundary consistency as needed.

This architectural decoupling enables robust low-latency (≈5 FPS for vision–language, >100 FPS AR rendering) egocentric pose estimation in hybrid environments. Notably, monolithic (non-segmented) pipelines failed to initialize on 5/9 test sequences (Lai et al., 24 Aug 2025).

4. LLM-Based AR Overlay Recommendation

SEER-VAR incorporates a GPT-based (GPT-4 / o4-mini) recommendation module, triggered upon contextual events such as low fuel, dashboard occlusion, or novel environmental cues:

• Prompt Structure: Chain-of-thought style system/user prompts specify (1) context identification, (2) vehicle telemetry, (3) segmented view images, followed by explicit questions on overlay content and 2D anchor bounding boxes.
• Output: The LLM returns a JSON payload designating overlay labels and bounding boxes in normalized image coordinates. Example outputs include $${}^{in}\!T_{C_{t_i}}, {}^{ext}\!T_{C_{t_i}} \in SE(3)$$2
• Prompt Engineering: Explicitly chaining context classification, content proposal, and anchoring in the prompt produces stable, contextually relevant recommendations (see Supplementary Sec. 8 in (Lai et al., 24 Aug 2025)).
• Performance: The agent achieves ~5 FPS on an RTX 4080.

Structured prompting enables robust, semantically appropriate overlays tightly bound to both intra and extra contexts.

5. EgoSLAM-Drive Dataset

Evaluation and training are facilitated via EgoSLAM-Drive, a multi-modal egocentric driving dataset:

• Data Modalities: Synchronized 1408×1408 RGB, stereo-style estimated depth, 6-DoF ground-truth (ArUco markers and high-rate IMU), AR annotation overlays, and context tags.
• Scope: Nine sequences in garages, parking lots, streets, intercity, and highways (1K–6K frames each); two vehicle types for dashboard diversity.
• Synchronization: Nanosecond-aligned timestamps across all modalities; consistent per-frame indices.
• Annotations: Segmentation masks (RGBA), AR overlay bbox labels (JSON), and context class labels. Privacy is ensured by offline blurring of faces and license plates.

This dataset establishes a benchmark for egocentric SLAM and AR in heterogeneous cabin–road environments (Lai et al., 24 Aug 2025).

6. Evaluation Methodology and Quantitative Results

SEER-VAR’s performance is evaluated across geometric, perceptual, and subjective axes:

• Spatial Consistency: Reprojection error measured as $\mathcal{X}(t_i)$8 over ArUco marker corners and all frames; mean ± std reported.
• Absolute Trajectory Error (ATE):

$\mathcal{X}(t_i)$9

• Perceptual Fidelity: LPIPS and NIQE of AR-augmented frames; lower values indicate higher fidelity.
• Ablation Studies: Comparison with/without loop closure detection and depth-guided segmentation.
• User Study: 176 licensed drivers rated AR-augmented sequences on (i) effort reduction, (ii) contextual relevance, (iii) anchoring accuracy, (iv) motion realism; $${}^{in}\!T_{C_{t_i}}, {}^{ext}\!T_{C_{t_i}} \in SE(3)$$0 vs. neutral across all metrics.

Key quantitative results are summarized below:

Metric	Intra w/o LCD	Intra w/ LCD	Extra w/o LCD	Extra w/ LCD
Reprojection Error (px)	1.22 ± 0.46	1.03 ± 0.40	0.66 ± 0.25	0.90 ± 0.36

AR Type	LPIPS	NIQE
Dashboard	0.040 ± 0.007	9.05 ± 0.61
Service Ad	0.029 ± 0.016	9.30 ± 0.42
Parking Info	0.060 ± 0.095	9.16 ± 0.71
Navigation Hint	0.062 ± 0.073	9.26 ± 0.85

Disabling depth-guided segmentation caused initialization failures on the majority of sequences, indicating its necessity. All overlay types yielded LPIPS ≤ 0.06 and NIQE ≈ 9, consonant with high perceptual alignment. User studies confirmed significant improvements in perceived realism, contextual appropriateness, and overlay usability ($${}^{in}\!T_{C_{t_i}}, {}^{ext}\!T_{C_{t_i}} \in SE(3)$$1) (Lai et al., 24 Aug 2025).

7. Significance and Impact

SEER-VAR demonstrates that explicit decoupling of cabin and road contexts via depth-guided semantic segmentation is critical for spatial stability and usability in mixed egocentric driving AR. The architecture’s dual SLAM branches (CASB) and structured LLM prompting establish a methodological foundation for robust, context-aware pose tracking and semantically relevant AR recommendation.

Key findings include:

• Consistent sub-pixel to low-pixel spatial alignment enabling tightly registered AR overlays.
• Low-latency, robust pose tracking at framerates compatible with real-time AR rendering.
• LLM-based overlay recommendation delivers contextually meaningful, perceptually coherent augmentation in both subjective and objective terms.
• The EgoSLAM-Drive dataset enables systematic benchmarking of egocentric SLAM and AR under challenging real-world driving conditions.

A plausible implication is that the architectural principles introduced in SEER-VAR can generalize to other domains requiring multi-context egocentric tracking, such as robotics, industrial AR, or human–computer interaction in dynamic settings (Lai et al., 24 Aug 2025).