GazeProphet: VR Gaze Prediction

• GazeProphet is a software-only VR gaze prediction system that forecasts a normalized 2D gaze point and confidence score using scene content and recent gaze history.
• It employs a Spherical Vision Transformer and an LSTM-based temporal encoder, fused adaptively, to achieve a median angular error of 3.83° in gaze prediction.
• The system outputs an uncertainty-aware gaze coordinate for foveated rendering, reducing dependency on dedicated eye-tracking hardware and lowering computational costs.

Searching arXiv for GazeProphet and closely related VR gaze prediction papers. GazeProphet is a software-only gaze prediction system for virtual reality that is designed to support foveated rendering without dedicated eye-tracking hardware. In the paper "GazeProphet: Software-Only Gaze Prediction for VR Foveated Rendering" (Ebadulla et al., 19 Aug 2025), the method predicts a single future gaze location as normalized 2D coordinates $(x,y)$ on the equirectangular representation of a 360-degree scene, together with a scalar confidence score indicating reliability. Its central premise is that foveated rendering can reduce computational demand by concentrating rendering quality near the user’s gaze, but hardware eye tracking remains a barrier because it increases cost, consumes power, adds integration complexity, requires calibration, and is available only on premium devices. GazeProphet addresses that constraint by combining a Spherical Vision Transformer for 360-degree scene analysis, an LSTM-based temporal encoder for recent gaze dynamics, and a multimodal fusion network that adaptively combines both sources of evidence (Ebadulla et al., 19 Aug 2025).

1. Definition and task scope

GazeProphet is formulated as a future gaze point predictor for immersive VR scenes rather than as an eye-image-based gaze estimator, saliency-map predictor, fixation heatmap generator, or full probabilistic scanpath model. The output is pointwise and uncertainty-aware: a single normalized 2D coordinate in equirectangular image space plus a confidence value in $[0,1]$ (Ebadulla et al., 19 Aug 2025). The coordinate prediction is later evaluated as angular distance on the sphere, which reflects the 360-degree viewing geometry more appropriately than planar image error alone.

This design choice distinguishes GazeProphet from several adjacent research threads. Appearance-based 3D gaze estimation methods such as CrossGaze predict a normalized 3D gaze vector from RGB images of the face and eyes, achieving a mean angular error of $9.94^\circ$ on Gaze360 in its pretrained setting (Cătrună et al., 2024). By contrast, GazeProphet assumes access to the current scene image and a short gaze history, and predicts the next gaze point within a VR panorama rather than inferring gaze from ocular appearance. A plausible implication is that GazeProphet belongs more naturally to the family of attention forecasting systems than to classical webcam- or headset-camera gaze estimators.

The paper also explicitly differentiates its objective from full-distribution modeling. There is no output saliency map, no discrete classification over spherical cells, and no full uncertainty distribution over gaze locations (Ebadulla et al., 19 Aug 2025). This makes the model comparatively direct for downstream rendering control, since a renderer needs a center point for the high-resolution region and a reliability estimate for deciding how aggressively to trust it.

2. Architectural organization

GazeProphet has three components operating in parallel: a Spherical Vision Transformer for spatial analysis of the 360-degree scene, an LSTM-based temporal encoder for recent gaze dynamics, and a multimodal fusion network that emits the final coordinate prediction and confidence (Ebadulla et al., 19 Aug 2025). The architecture is therefore explicitly multimodal, with scene content and temporal gaze behavior treated as complementary signals.

The spatial branch begins from a 360-degree VR scene image represented in equirectangular projection at resolution $256 \times 512$. The image is divided into $16 \times 16$ pixel patches, producing a $16 \times 32$ grid and thus 512 tokens. Each patch is linearly projected into a 384-dimensional embedding (Ebadulla et al., 19 Aug 2025). Standard ViTs assume planar image geometry, but equirectangular images have latitude-dependent distortion, so the paper states that GazeProphet applies patch-wise normalization to compensate for varying pixel density across latitude, although the exact normalization formula is not given.

The main spherical-awareness mechanism is the positional encoding. For patch location $(i,j)$, the model maps positions to spherical coordinates according to

$$\theta = \frac{j \pi}{32}, \quad \phi = \frac{(i-8)\pi}{16}.$$

These coordinates are encoded using spherical harmonics $Y_l^m(\theta,\phi)$ up to degree $l=4$, producing 25 harmonic coefficients per patch position, which are then projected to 384 dimensions (Ebadulla et al., 19 Aug 2025). The transformer stack uses 6 layers with 8 attention heads, so each head has $[0,1]$0. The attention weight is the standard scaled dot-product form: $[0,1]$1 Each block is followed by layer normalization and a feed-forward network with hidden dimension 1536, using GELU activation and dropout $[0,1]$2. After the final layer, global average pooling over the 512 token embeddings yields a 384-dimensional scene representation (Ebadulla et al., 19 Aug 2025).

The temporal branch consumes a sequence of 10 previous gaze points. Each gaze point is described as normalized $[0,1]$3 coordinates and a confidence value, forming a 3D vector per timestep, while time deltas between consecutive gaze points are also said to be used as temporal context and are log-scaled (Ebadulla et al., 19 Aug 2025). The temporal encoder is an LSTM with 128 hidden units and standard gating dynamics: $[0,1]$4 Rather than using only the final hidden state, the model adds temporal attention across all 10 timesteps: $[0,1]$5 The output is a 128-dimensional summary of recent gaze behavior (Ebadulla et al., 19 Aug 2025).

The multimodal fusion network combines the 384-dimensional spatial vector and the 128-dimensional temporal vector. The paper first states that concatenation produces a 512-dimensional representation projected to 256 dimensions, then defines adaptive fusion weights

$[0,1]$6

and a fused representation

$[0,1]$7

The text contains a dimensional inconsistency because $[0,1]$8 is 384D, $[0,1]$9 is 128D, and $9.94^\circ$0 is described as 256D (Ebadulla et al., 19 Aug 2025). The most paper-grounded interpretation is that both branches are projected into a common latent space before weighted combination, but the exact projection equations are not provided.

3. Outputs, loss design, and confidence calibration

From the fused 256-dimensional representation, GazeProphet uses two output heads. The gaze head is a two-layer MLP with 128 hidden units and ReLU activation, followed by sigmoid output so that the predicted coordinates remain in $9.94^\circ$1. The confidence head uses a similar two-layer MLP and sigmoid output to produce a scalar confidence $9.94^\circ$2 (Ebadulla et al., 19 Aug 2025).

The total loss is

$9.94^\circ$3

with $9.94^\circ$4 and $9.94^\circ$5 (Ebadulla et al., 19 Aug 2025). The gaze regression term is mean squared error: $9.94^\circ$6 The confidence term is also mean squared error, but the target is a binary correctness indicator: $9.94^\circ$7 Here $9.94^\circ$8, which the paper states corresponds to approximately 10 pixels on a $9.94^\circ$9 equirectangular image (Ebadulla et al., 19 Aug 2025). Confidence therefore estimates whether the prediction error is small enough to be acceptable for downstream use. The paper also reports that alternative loss weight ratios such as 1:1 or 5:1 were inferior in validation, either degrading gaze accuracy or worsening confidence calibration (Ebadulla et al., 19 Aug 2025).

This confidence design is narrower than full probabilistic uncertainty modeling. The paper does not use Gaussian negative log-likelihood, quantile loss, or evidential uncertainty objectives (Ebadulla et al., 19 Aug 2025). A plausible implication is that the confidence output is intended for operational decision-making in a rendering pipeline rather than for calibrated Bayesian uncertainty estimation in the stricter sense.

4. Experimental setting and empirical performance

The experiments use the Sitzmann VR Saliency Dataset, described as containing eye-tracking recordings from multiple users viewing diverse VR environments represented as equirectangular images (Ebadulla et al., 19 Aug 2025). The paper does not provide exact numbers for users, scenes, sequences, or train/validation/test partitioning, which limits reproducibility.

The main evaluation metric is median angular error on the sphere, reported as $256 \times 512$0 (Ebadulla et al., 19 Aug 2025). Additional reported metrics are MSE in normalized image coordinates, accuracy within pixel thresholds such as 10, 20, and 50 pixels, and confidence statistics. The paper does not explicitly print the spherical angular-distance formula, but the evaluation is described as converting coordinate prediction error into angular distance on the sphere (Ebadulla et al., 19 Aug 2025).

The principal quantitative comparison is summarized below.

Model	MSE	Angular error	Acc@10px
GazeProphet	0.0035	$256 \times 512$1	67.2%
Temporal-Only	0.0090	$256 \times 512$2	45.8%
Spatial-Only	0.0508	$256 \times 512$3	28.3%
DeepGaze-VR	0.0421	$256 \times 512$4	31.7%

GazeProphet also reports average confidence values: 0.997 for the full model, 0.562 for Temporal-Only, 0.555 for Spatial-Only, and 0.487 for DeepGaze-VR (Ebadulla et al., 19 Aug 2025). The paper’s headline claim is that the median angular error of $256 \times 512$5 is a 24% improvement over the best saliency-based baseline and a 41% improvement in angular error over the temporal-only baseline (Ebadulla et al., 19 Aug 2025). The authors also state that paired $256 \times 512$6-tests yield $256 \times 512$7 in each comparison and that Cohen’s $256 \times 512$8 exceeds 0.8 throughout; specifically, the comparison with Temporal-Only gives $256 \times 512$9, and the comparison with Spatial-Only gives $16 \times 16$0. The 95% confidence interval for angular error is reported as $16 \times 16$1 (Ebadulla et al., 19 Aug 2025).

The confidence analysis is intentionally operational rather than exhaustive. The paper reports an average confidence of 0.997 and states that predictions with confidence above 0.9 achieve 97.1% accuracy (Ebadulla et al., 19 Aug 2025). Lower-confidence predictions correspond to lower observed performance. At the same time, the paper does not report standard calibration metrics such as Expected Calibration Error, Brier score, reliability diagrams, or NLL (Ebadulla et al., 19 Aug 2025). This suggests that the confidence mechanism is validated mainly as a policy-control variable for rendering rather than as a general uncertainty benchmark.

5. Spatial behavior, ablations, and rendering implications

A notable empirical claim is that GazeProphet avoids the center bias common in saliency-like methods. Its center-region angular error is $16 \times 16$2, while peripheral-region angular error is $16 \times 16$3, indicating nearly uniform performance over the visual field (Ebadulla et al., 19 Aug 2025). By contrast, the baselines perform better in central regions and degrade toward the periphery. In 360-degree VR, this matters because users can look anywhere, so a predictor that implicitly assumes center preference is structurally mismatched to the task.

The ablation findings reinforce the contribution of each component. Removing the spherical Vision Transformer raises angular error to $16 \times 16$4, eliminating the LSTM temporal encoder increases error to $16 \times 16$5, and replacing learned multimodal fusion with simple concatenation worsens error by $16 \times 16$6 (Ebadulla et al., 19 Aug 2025). The ablations therefore support a hierarchy in which temporal gaze history is especially important, scene understanding is materially beneficial, and adaptive fusion is superior to naive feature combination.

For rendering, the intended use is explicit. The predicted gaze coordinate determines the center of the high-resolution region, while the confidence score informs how aggressively the renderer should trust the prediction (Ebadulla et al., 19 Aug 2025). In high-confidence cases, stronger foveation can save more computation; in lower-confidence cases, the renderer can fall back to a more conservative profile, such as enlarging the high-quality region or reducing peripheral degradation. This makes the uncertainty-aware output directly operational.

However, the paper gives limited runtime detail. It argues for practical deployment feasibility and identifies real-time optimization as future work, specifically mentioning the need to reduce latency below 10 ms for seamless foveated rendering (Ebadulla et al., 19 Aug 2025). No FLOPs, parameter count, FPS, inference timing, or hardware deployment profile is reported. This suggests that the rendering motivation is strong, but the real-time systems claim remains incompletely substantiated.

6. Position within the broader gaze-prediction literature

GazeProphet sits at an intersection of several strands of research. Compared with appearance-based gaze estimation methods such as CrossGaze (Cătrună et al., 2024), "Recurrent CNN for 3D Gaze Estimation using Appearance and Shape Cues" (Palmero et al., 2018), or DHECA-SuperGaze (Šikić et al., 13 May 2025), it does not infer gaze from eye or face imagery. Compared with egocentric fixation models such as "Unsupervised Gaze Prediction in Egocentric Videos by Energy-based Surprise Modeling" (Aakur et al., 2020), it is not primarily an unsupervised current-fixation localizer. Compared with long-horizon generative models such as "Infinite Gaze Generation for Videos with Autoregressive Diffusion" (Kang et al., 26 Mar 2026), it is not a raw trajectory generator over arbitrary-length video. Its niche is short-horizon, software-only gaze-point prediction in panoramic VR scenes for a control application, namely foveated rendering (Ebadulla et al., 19 Aug 2025).

The closest lineage within the supplied literature is GazeProphetV2 (Ebadulla et al., 25 Nov 2025), which is presented as a successor emphasizing multimodal integration of gaze history, head movement, and scene content for mobile VR. GazeProphetV2 uses a three-branch CNN+LSTM architecture with gated fusion and predicts 1-, 2-, and 3-frame future gaze coordinates from 15-step histories of gaze and head orientation plus the current scene image (Ebadulla et al., 25 Nov 2025). It reports cross-scene evaluation on 22 VR scenes with 5.3M gaze samples and an overall reported accuracy of 93.1%, although the exact definition of that accuracy metric is not specified (Ebadulla et al., 25 Nov 2025). This suggests a development from the original GazeProphet’s scene-plus-gaze-history formulation toward a broader multimodal VR forecasting framework that explicitly includes head dynamics.

A plausible interpretation of this progression is that software-only VR gaze prediction is moving from saliency-style scene reasoning toward multimodal behavioral forecasting. In that sense, GazeProphet occupies an intermediate point: it already uses scene and temporal gaze history jointly, but it remains a one-step point predictor with scalar confidence rather than a full sequence model or multimodal social-behavioral predictor.

7. Limitations and open issues

Several limitations are explicit in the paper. First, GazeProphet depends on prior gaze history at inference time: it consumes the previous 10 gaze points plus timing information (Ebadulla et al., 19 Aug 2025). It is therefore software-only in the sense of not requiring dedicated eye-tracking hardware, but it is not zero-history prediction. The paper does not specify a bootstrap mechanism for startup or recovery after interruptions.

Second, the output is only a single point and a scalar confidence, not a full spatial distribution (Ebadulla et al., 19 Aug 2025). This simplifies rendering control, but it may be brittle in ambiguous scenes where multiple targets are plausible. Third, the representation remains rooted in equirectangular images with distortion-aware normalization and spherical-harmonic positional encoding rather than fully spherical attention or geometry-preserving convolutions (Ebadulla et al., 19 Aug 2025). Fourth, the training setup omits optimizer, learning rate, batch size, epoch count, hardware, and implementation framework, which creates significant reproducibility gaps (Ebadulla et al., 19 Aug 2025).

Finally, software prediction may not fully substitute for hardware eye tracking in applications demanding extremely low latency or sub-degree precision, such as precise UI selection, vision science studies, or very aggressive foveation (Ebadulla et al., 19 Aug 2025). This limitation is consistent with the broader literature: systems optimized for unconstrained appearance-based gaze estimation or smartphone point-of-gaze prediction still emphasize generalization, robustness, or low annotation cost rather than full replacement of high-end hardware in the most demanding regimes (Cătrună et al., 2024, Guo et al., 2019, Ghosh et al., 2022).

Taken together, these constraints define GazeProphet as a VR-specific, uncertainty-aware, multimodal point predictor whose novelty lies in adapting transformer-based scene encoding to spherical panoramas and coupling it with recent gaze history for software-only foveated rendering control (Ebadulla et al., 19 Aug 2025). Its reported $16 \times 16$7 median angular error, strong gains over spatial-only and temporal-only baselines, and near-uniform central-versus-peripheral behavior indicate that software-only gaze prediction can be sufficiently accurate to support practical VR rendering policies, even if it does not eliminate the need for hardware tracking in every application (Ebadulla et al., 19 Aug 2025).