AutoGaze: Automated Gaze Estimation and Attention Mapping

Updated 2 July 2026

AutoGaze is a comprehensive framework that automatically estimates and predicts human gaze using methods like Gaussian Process Regression and deep neural networks.
It integrates multi-modal inputs—from spatiotemporal video and egocentric cues to 3D SLAM reconstructions—to enhance applications in driver assistance, AR/VR, and HCI.
AutoGaze optimizes token selection and real-time calibration to significantly improve prediction accuracy and system efficiency in diverse operational settings.

AutoGaze refers to a diverse set of frameworks, algorithms, and systems for automatically inferring, predicting, tracking, or calibrating human gaze and visual attention in digital environments. The term encompasses probabilistic modeling for driver monitoring, deep learning for saliency prediction, efficient spatiotemporal patch selection in large-scale video understanding, online egocentric gaze estimation, self-calibrating assistive technologies, and real-time 3D reconstruction of human attention in physical spaces. Common to all incarnations is the goal of leveraging vision-based, behavioral, or physiological signals to map, anticipate, or optimize what and where a person (or artificial system) is attending, often for safety, efficiency, or human-computer interaction enhancement.

1. Probabilistic Modeling of Gaze and Salient Regions

One instantiation of AutoGaze is the probabilistic inference of driver gaze regions from head pose using Gaussian Process Regression (GPR). A 6D head-pose vector $x \in \mathbb{R}^6$ , comprising 3D position and three orientation angles, is mapped to the conditional gaze distribution $P(y|x)$ , where $y$ is a unit vector on the sphere or parameterized by gaze angles $(\theta, \phi)$ . The system constructs (1−α)-confidence regions $C_{1-\alpha}(x)$ :

$\int_{y \in C_{1-\alpha}(x)} P(y | x) \, dy = 1-\alpha$

GPR models for $\theta$ and $\phi$ are trained on naturalistic driving data, with uncertainty modeled directly from data density and head-gaze ambiguity. The posterior at a query head pose $x_*$ yields a 2D Gaussian ellipse in angular space. On the UTDrive dataset, GPR achieves 95% gaze coverage within 3.77% of the sphere, outperforming linear regression (4.74%) and neural nets (5.39%), and showing close empirical-theoretical coverage agreement (Jha et al., 2020).

2. Deep Learning for Attention and Saliency in Driving

AutoGaze systems in the driver-assistance context extend to spatiotemporal saliency prediction from video. The framework of Palazzi et al. utilizes a two-stage convolutional architecture operating on 16-frame roof-mounted driving clips to predict continuous-valued attentional maps $G(x, y) \in [0,1]$ representing gaze probability distributions. The architecture comprises a 3D CNN backbone for coarse spatiotemporal features, followed by a 2D convolutional refinement stream. Training uses mean squared error, evaluated via correlation coefficient (CC) and KL divergence against ground-truth attentional maps.

On the DR(eye)VE dataset, AutoGaze achieves CC = 0.55 and KL = 1.42, consistently outperforming prior saliency models, especially in challenging, less-predictable driving frames. The system supports an assisted-driving paradigm where the predicted attention overlay can guide, but not override, driver attention (Palazzi et al., 2016).

A further evolution, SAGE-Net, augments gaze prediction with scene semantics, per-pixel segmentation masks, Monodepth2-derived depth, ego-vehicle speed, and pedestrian intent. By combining softmax-attention between semantic features and ground-truth gaze maps, modulating by depth and intent signals, SAGE-Net yields an 8.5% AUC improvement and substantial F1/KL gains in critical scenarios. This multi-modal integration addresses center-bias and enhances attention prediction for salient objects (cars, pedestrians) (Pal et al., 2019).

3. Online Egocentric Gaze Estimation via Autoregressive Modeling

In egocentric settings, where head/eye pose is absent, AutoGaze is formulated as strictly causal, sequential autoregressive decoding. At each timestep, the model predicts the spatial heatmap $P(y|x)$ 0 over the current frame $P(y|x)$ 1 conditioned on a fixed-length window $P(y|x)$ 2 and current visual tokens:

$P(y|x)$ 3

A ViT backbone encodes frames and a template crop around previous gaze centers. History heatmaps are tokenized via 1×1 convolution, embedded with temporal and token-type positions, and fused with visual tokens in a masked transformer decoder. The KL divergence loss supervises outputs, and scheduled sampling bridges train/test exposure bias. On EGTEA, Ego4D, and EgoExo4D, ARGaze sets new F1 state-of-the-art (e.g., EGTEA: 44.48 F1 vs. 41.57 for the strongest baseline), while maintaining 11 ms latency at 89 FPS with strictly bounded memory (Li et al., 4 Feb 2026).

4. Efficiency and Scalability: Autoregressive Gazing for Video Understanding

For large-scale video QA, AutoGaze is implemented as an autoregressive, multi-scale patch selection module that sits ahead of Vision Transformers (ViTs) in multi-modal LLM or video pipelines. The Conv-Transformer-based gazing module sequentially selects patches conditioned on past history and minimizes the number needed to ensure that a reconstruction autoencoder (e.g., VideoMAE) reaches error $P(y|x)$ 4:

$P(y|x)$ 5

It is trained via next-token prediction on approximately optimal patch sequences and refined via on-policy group reinforcement learning (GRPO) for improved downstream task reward. The module reduces tokens by up to 100× and accelerates ViTs/MLLMs up to 19× with negligible (<0.5%) QA drop. In high-resolution (4K), 5-minute HLVid benchmarks, AutoGaze scales MLLMs to 1024 frames and outperforms the previous best by +4.5% (Shi et al., 12 Mar 2026). Unlike in- or post-ViT pruning, AutoGaze eliminates redundant pixels before ViT tokenization, providing major advantages for memory- and compute-limited settings.

5. AutoCalibrating and Self-Optimizing Gaze Systems in HCI and VR

AutoGaze principles also figure in real-time, user-facing calibration and drift correction. In the EyeO system, the system discriminates between user "reading" (perception, no compensation) and "typing" (action, with compensation) modes. During reading, fixations within a calibration zone update an additive offset $P(y|x)$ 6 to correct the mapping from raw gaze to screen coordinates. The update uses a sliding window mean residual, clipped to ensure stability:

$P(y|x)$ 7

Enabling seamless online calibration significantly improves eye-typing performance: mean typing speed increases from 20.31 to 22.73 CPM ( $P(y|x)$ 8), and abort frequency drops from 0.60 to 0.05 ( $P(y|x)$ 9). EyeO is especially beneficial for populations with frequent calibration loss (e.g., ALS users), reducing manual recalibration burden and effort (Saran et al., 2023).

6. 3D Gaze Recovery and Attention Mapping in Real Environments

AutoGaze has also been implemented as a real-time pipeline for fusing wearable eye-tracking (ETG) with RGB-D SLAM to reconstruct user gaze and attention in 3D environments. ETG scene video frames are localized in the SLAM-generated mesh via SIFT keypoints and a Perspective-n-Point solver. 2D gaze points are back-projected through system calibration matrices to construct a gaze ray in world coordinates:

$y$ 0

Ray-mesh intersection yields fixation points in the mesh, which are aggregated for heatmap generation and AOI dwell counting. The pipeline supports automatic 3D annotation of ROIs—a critical feature for integrated semantic analysis of user attention in real or virtual environments. Reported accuracy is 0.5–1.1 cm in 3D at mean angular errors of 0.6°, with >90% reliability for semantic ROI mapping (Paletta et al., 2013, Paletta et al., 2013).

7. Automated AOI Annotation and Attention Analytics at Scale

Recent tools such as Gaze2AOI generalize AutoGaze principles for automated mapping of gaze to detected object AOIs in video. Using YOLOv8 with ByteTrack, individual AOI instances are recognized and temporally tracked. Each gaze sample is mapped to AOIs via geometric inclusion. The tool computes standard metrics—time-to-first-fixation (TTFF), dwell time, revisit frequency—across up to 600 object classes, and supports interactive visualization and refinement. This facilitates scalable, reproducible analytics, especially in behavioral and user-experience research. Current limitations include absence of formal accuracy benchmarks and preclusion of real-time streaming, with future extensibility towards domain-specific fine-tuning and more complex semantic hierarchies (Trajkovska et al., 2024).

AutoGaze encompasses a spectrum of systems and methodologies for automated, real-time, and context-aware gaze estimation, calibration, and attention analytics, with demonstrable impact in driver assistance, assistive technologies, video QA, egocentric AR/VR, and human-attention mapping in complex scenes. Architectures span from GPR and random forests to deep neural attention and autoregressive transformers; evaluation is domain-specific but consistently emphasizes real-time, high-accuracy, and user-adaptive performance. The underlying theme is the efficient, robust mapping of behavioral or visual signals to actionable estimations of attention, compatible with large-scale deployment and increasingly complex interaction modalities.