Egocentric 3D Visual Span Forecasting

• Egocentric 3D Visual Span Forecasting is the study of predicting future volumetric regions of visual attention from egocentric sensor data using geometric and behavioral modeling.
• It involves lifting gaze directions into sparse 3D occupancy grids through keypoint filtering, local frame transformations, and integration with encoder-decoder architectures like 3D U-Net combined with Transformers.
• Applications in AR/VR, assistive navigation, and robotics enable proactive resource allocation and hazard detection, despite challenges in modeling multi-modal attention under dynamic conditions.

Egocentric 3D Visual Span Forecasting encompasses the paper and prediction of where a person’s visual perception will be focused within their immediate three-dimensional environment, grounded in wearable egocentric sensors such as video, IMU, and SLAM-derived geometry. This field aims to forecast not only head pose and gaze direction but also the full 3D “visual span”—the volumetric regions in space that are likely to occupy the individual’s future field of view. The task interlaces geometric, semantic, and behavioral modeling and has direct implications for AR/VR, assistive navigation, scene understanding, and robot autonomy (Yun et al., 23 Nov 2025).

1. Formal Problem Definition

Egocentric 3D visual span forecasting is defined as follows: Given a temporal sequence of egocentric video frames, sensor data (IMU, SLAM), and estimated head/gaze orientation, the objective is to predict future 3D volumes that will be visually attended. Let $\mathcal{P} = \{(p_i, \sigma_i, t_i)\}$ denote SLAM keypoints in $\mathbb{R}^3$, $\mathcal{E} = \{E_t\}$ the sequence of head poses $E_t \in \mathrm{SE}(3)$, and $\mathbf{g}_t \in \mathbb{R}^3$ the gaze direction. At each instant $t$, the instantaneous gaze is lifted into a 3D occupancy grid $V_t \in \{0,1\}^{4\times R\times R\times R}$ partitioned by angular eccentricities (e.g., orientation/frustum, near-periphery, central, foveal). The forecasting function $f$ maps a window of past $V_t$ to a future union of spans:

$$f:\,\{V_{t-T_p+1},\dots,V_t\} \longmapsto \widetilde{Y} \approx \bigcup_{\tau = t+1}^{t+T_f} V_\tau \in \{0,1\}^{4\times R\times R\times R}$$

This reframes traditional 2D gaze anticipation into a fully 3D spatial forecasting task, emphasizing spatial continuity, semantic scene context, and explicit temporal aggregation (Yun et al., 23 Nov 2025).

2. 3D Visual Span Representation and Gaze Lifting

Conversion from gaze direction to 3D volumetric spans involves multiple steps:

• Keypoint Filtering: $p_i \in \mathcal{P}_t$ are selected based on proximity to camera center $t_t$, a spatial threshold $D/2$, and neighbor-based outlier rejection.
• Transformation to Local Frame: Points are converted to the local camera-centric coordinate frame by $p_i^{\mathrm{loc}} = E_t^{-1} p_i$.
• Gaze-Cone Classification: For each angular eccentricity $\theta$, SLAM points are classified as inside the gaze cone if their cosine angle with $\mathbf{g}_t$ exceeds $\cos\theta$:

$$Q_t^{\theta} = \Bigl\{p_i \in \mathcal{P}_t \;|\; \frac{\langle p_i^{\rm loc},\,\mathbf{g}_t\rangle}{\|p_i^{\rm loc}\| \|\mathbf{g}_t\|} > \cos\theta \Bigr\}$$

• Volumetric Occupancy Encoding: The gaze span for each $\theta$ is accumulated into a sparse occupancy grid, indexed temporally and by span class:

$$V_{[t_b,t_e]}^{\theta}(i,j,k) = \mathcal{I}\!\Bigl(\left|\left\{p \in \bigcup Q_t^{\theta} \textrm{ in voxel } (i,j,k)\right\}\right|>0\Bigr)$$

Stacking for multiple angular levels yields a $4 \times R \times R \times R$ binary mask per frame (Yun et al., 23 Nov 2025).

3. Model Architectures for 3D Span Forecasting

The dominant paradigm is to unify spatiotemporal reasoning across both geometry and attention with encoder-decoder architectures:

• EgoSpanLift (3D U-Net + Transformer):
  • Input Representation: Past sequence of $T_p$ 5-channel $R^3$ tensors ($4$ span levels + 1 background occupancy).
  • 3D U-Net Encoder: Captures fine-grained spatiotemporal semantics via repeated $3\times3\times3$ convolutions, spatially pooling to per-timestep embeddings $v_1, \dots, v_{T_p}$.
  • Global Temporal Embedding: Aggregates temporal sequence into a global representation $v_{\rm head}$.
  • Unidirectional Transformer: Applies causal masked self-attention across past timesteps plus the aggregated vector.
  • 3D U-Net Decoder: Reconstructs the predicted future span $$\widetilde{Y} \in [0,1]^{4 \times R \times R \times R}$$ using transposed convolutions and skip-connections.
  • Training Loss: Supervision is applied to $\widetilde{Y}$ with a soft Dice (F1) loss:

  $$\mathcal{L}_{\rm dice} = 1 - \frac{2\,\sum_{c,i,j,k} \widetilde{Y}_{c,i,j,k} Y_{c,i,j,k}}{\sum_{c,i,j,k} \widetilde{Y}_{c,i,j,k} + \sum_{c,i,j,k} Y_{c,i,j,k} + 1}$$

  Dice loss is favored for sparse occupancy tasks (Yun et al., 23 Nov 2025).

• Pose Forecasting Models: Some approaches focus on explicit 6D head pose or full-body pose trajectory forecasting, as in LookOut and UniEgoMotion (Pan et al., 20 Aug 2025, Patel et al., 2 Aug 2025), to enable indirect visual span anticipation by “rolling out” predicted frustums.

4. Datasets and Benchmarks

Three major resources enable benchmarking of egocentric 3D visual span forecasting:

• FoVS-Aria: Sourced from Aria Everyday Activities, comprising 23.2K samples (past 2s → union-of-next-2s); $D = 3.2$ m cube, $R=16$.
• FoVS-EgoExo: Sourced from Ego-Exo4D, with 341.4K samples (past 4s → next 4s). Activities span cooking, music, medical tasks, repair, and bouldering.
• Aria Navigation Dataset (AND): Project Aria glasses, 4 hours in 18 scenes, focusing on navigation and head-pose prediction for robot planning (Pan et al., 20 Aug 2025).

Label construction always includes fine-grained volumetric span masks across eccentricities (orientation/frustum, near-periphery, central, foveal) and temporally pooled future targets.

5. Evaluation Metrics and Quantitative Results

Metrics are span-level intersection-over-union (IoU), F1 on occupancy, and Euclidean centroid error for 3D foveal prediction. Direct comparison with adapted 2D and 3D baselines is standard.

Table 1. 3D IoU & F1 on FoVS-Aria (Test Split) (Yun et al., 23 Nov 2025) | Method | Ori IoU | Per IoU | Cen IoU | Fov IoU | |----------------|---------|---------|---------|---------| | EgoChoir | 0.4959 | 0.4302 | 0.2612 | 0.1987 | | EgoSpanLift | 0.5838 | 0.4886 | 0.3513 | 0.2836 |

Table 2. Foveal‐Span Centroid Error (cm):

Method	min	avg	max
GLC-based	59.7	73.5	87.2
Ours	19.0	34.9	51.2

On FoVS-EgoExo, EgoSpanLift yields test 3D IoUs: Ori 0.5230, Per 0.5108, Cen 0.4212, Fov 0.3692. Projecting predicted foveal region back to the 2D image achieves F1 = 0.515, matching 2D-trained gaze anticipation methods without 2D-specific supervision (Yun et al., 23 Nov 2025).

Pose-based forecasting approaches perform trajectory-level evaluation via L₁ translation/rotation, non-collision ratios (Pan et al., 20 Aug 2025), and, for full-body prediction, MPJPE, MPJPE-PA, Foot Contact, and semantic similarity (Patel et al., 2 Aug 2025).

6. Applications, Limitations, and Future Directions

• AR/VR and Foveated Rendering: Proactive high-resolution rendering within future spans reduces compute requirements by focusing resources where gaze is expected (Yun et al., 23 Nov 2025).
• Assistive Navigation and Robotics: Head/gaze span forecasting enables early hazard warnings and preemptive path planning or obstacle avoidance in both human-assistive and robotic agents (Pan et al., 20 Aug 2025).
• Limitations: Current deterministic regressors (e.g., in head pose forecasting) are limited in modeling multi-modal trajectory futures; performance drops when viewing behavior is highly ambiguous or intent-driven (Pan et al., 20 Aug 2025). Sparse SLAM constrains attention localization in cluttered scenes; 3D occupancy can be limited by scene coverage and head-pose estimation noise (Yun et al., 23 Nov 2025).
• Future Directions: Proposed advances include generative (diffusion-prior) approaches for multimodal trajectory anticipation, denser SLAM/geometry for finer volumetric spans, explicit integration of body-pose forecasting, and extending beyond vision to multi-sensory (e.g., auditory or proprioceptive) span prediction (Yun et al., 23 Nov 2025, Patel et al., 2 Aug 2025).

7. Relationship to Egocentric Pose and Motion Forecasting

Egocentric 3D visual span forecasting is closely related to, but conceptually distinct from, egocentric pose trajectory forecasting. While span forecasting predicts gaze-constrained volumetric attention, methods such as UniEgoMotion forecast the full 3D body motion (parameterized in SMPL-X), using head-centric canonicalization to align body pose to the camera frame. Diffusion-based models, self-attentive scene encodings (e.g., via ViT/DINOv2), and multimodal trajectory/interaction representation are extensively employed in both lines of research (Patel et al., 2 Aug 2025). A plausible implication is that future visual span forecasting models may directly benefit from continuous joint modeling of head, gaze, and body motion in a unified latent space, enabling more robust anticipation in dynamic real-world settings.