Gaze-Regularized Training Framework

• Gaze-Regularized Training Framework is a methodology that leverages eye gaze cues to guide model attention and improve generalization in visually grounded AI applications.
• It integrates gaze heatmaps and fixation data into network architectures using techniques like KL-divergence regularization to align model features with human perceptual priors.
• This approach enhances performance and interpretability in domains such as vision-language models, robotics, and imitation learning, promoting sample efficiency and robustness.

A gaze-regularized training framework is a methodological paradigm in machine learning that employs eye gaze cues—specifically gaze fixations, saccades, or heatmap distributions—as structured inductive biases during training, guiding model attention or feature learning toward task-relevant visual or spatio-temporal regions. This approach leverages human perceptual priors to improve generalization, interpretability, and robustness in tasks spanning vision-language modeling, imitation learning, robotics, and gaze estimation.

1. Foundational Principles and Motivation

Gaze-regularized training exploits the empirical alignment between human attention and salient or task-critical scene information. Eye gaze traces, available as spatial point sequences or temporally-aggregated heatmaps, encode strategic cues for intent, anticipation, and fine-grained action planning. Integrating these signals into learning pipelines addresses limitations of models that attend solely to raw visual data, which may result in diffuse or misaligned attention maps and reduced sample efficiency. This framework is applicable across diverse settings: egocentric activity recognition (Pani et al., 24 Mar 2026, Pani et al., 24 Oct 2025), vision-language-action (VLA) models for robotic manipulation (Pani et al., 24 Mar 2026), imitation learning via behavioral cloning (Thakur et al., 2021), and gaze estimation (Yin et al., 2024, Guo et al., 2019, Yang, 27 Nov 2025, Adebayo et al., 2024).

2. Computational Frameworks and Model Integration

Gaze-regularized training can be implemented via several architectural strategies:

• Vision-LLMs with Gaze-Alignment: Gaze-regularized VLMs (e.g., based on ViT and Flamingo architectures) insert "Gaze-Guided Attention Blocks" after the vision encoder. Keys and values are derived from image patch tokens, while gaze-based queries are generated by overlaying gaze heatmaps on RGB frames and passing them through the encoder. Attention outputs are then regularized against human gaze distributions via Kullback–Leibler (KL) divergence (Pani et al., 24 Mar 2026, Pani et al., 24 Oct 2025).
• Imitation and Action Learning: In Gaze Regularized Imitation Learning (GRIL), a dual-head network predicts both agent actions and human gaze locations from shared representations, with a multi-objective loss jointly optimizing behavioral cloning and gaze prediction. This encourages learned features to capture human-attended regions essential for sample-efficient policy acquisition (Thakur et al., 2021).
• Robotics and VLA Models: Robotic manipulation models use gaze predictors to synthesize heatmaps from demonstration video, aggregated and converted into patch-level distributions. These distributions regularize transformer attention heads via KL divergence, biasing the agent toward human-like focus without requiring gaze at inference (Pani et al., 24 Mar 2026).
• Gaze Estimation and Domain Generalization: Models such as CLIP-Gaze project network features away from language-defined "gaze-irrelevant" semantic subspaces using contrastive or cosine regularizers, and enforce structural consistency among samples through rank-based losses (Yin et al., 2024). HybridGazeNet and SLYKLatent employ geometric or statistical regularization terms—uncertainty-weighted losses, shape priors, or inverse explained-variance weighting—to down-weight high-uncertainty or noisy samples and focus model capacity on informative training regions (Adebayo et al., 2024, Guo et al., 2021).

3. Gaze Data Processing and Supervision Signals

Gaze-regularized pipelines depend critically on the representation and processing of gaze traces:

• Heatmap Construction: Raw eye-tracker outputs $\{(t, x_t, y_t)\}$ are temporally aggregated, filtered for occlusions (typically via dense optical flow checks), and converted to dense heatmaps. These heatmaps are often Gaussian-smoothed and normalized to produce patch-level distributions over image grids (Pani et al., 24 Mar 2026, Pani et al., 24 Oct 2025).
• Patch-Level Alignment: For transformer-based models, gaze heatmaps are partitioned into $N$ non-overlapping patches. The gaze mass per patch forms a probability distribution $\mathbf{H} \in \Delta^N$ to be used as a target for attention alignment.
• Gaze-Overlaid Images and Pseudo-Gaze: Gaze supervision can be embedded as visual overlays, i.e., $$G_T = \alpha\,\text{RGB}_T + (1-\alpha)\,\text{colormap}(\mathbf{H}_T)$$, which serve as model input. Alternatively, in inference-time gaze-free deployment, a synthetic heatmap predictor may be trained to regress $\mathbf{H}_T$ from $\text{RGB}_T$ alone, enabling test-time gaze regularization by proxy (Pani et al., 24 Mar 2026).
• Alternative Supervisory Signals: In gaze-based imitation learning, direct regression to 2D gaze points is used, while self-supervised frameworks may weight regression terms by inverse explained variance or learned per-sample uncertainty (Adebayo et al., 2024, Guo et al., 2021).

4. Regularization Objectives and Loss Function Design

The core of gaze-regularized training is the loss function incorporating gaze-derived terms:

• KL-Divergence Attention Regularizer: Let $\mathbf{A} \in \Delta^N$ denote the model's attention weights, and $\mathbf{H} \in \Delta^N$ the gaze-derived target. The regularization penalty is:

$$D_{KL}(\mathbf{A}\,\|\;\mathbf{H}) = \sum_{i=1}^N A_i \log(A_i / H_i)$$

The total training loss often sums a standard task loss (e.g., cross-entropy for text, MSE for actions) with the gaze regularizer, weighted by $\lambda$: $N$0 (Pani et al., 24 Mar 2026, Pani et al., 24 Oct 2025, Pani et al., 24 Mar 2026).

• Multi-Objective Behavioral Imitation: In GRIL,

$N$1

where both action and gaze prediction losses are calculated using mean-squared error (Thakur et al., 2021).

• Cosine and Distributional Regularization: Gaze estimation approaches (e.g., CLIP-Gaze, SLYKLatent) employ regularizers that enforce projection of relevant features away from language-defined distractors and utilize inverse variance weighting to combat aleatoric and domain-induced uncertainty (Yin et al., 2024, Adebayo et al., 2024).
• Shape and Geometric Constraints: In gaze tracking and geometric gaze estimation, shape-based penalties (e.g., ellipse fit error) and analytic reconstruction losses constrain predicted gaze vectors or segmentation boundaries, directly improving the semantic precision and generalization (Yang, 27 Nov 2025, Guo et al., 2021).

5. Empirical Outcomes and Quantitative Gains

Gaze-regularized frameworks consistently yield substantial performance improvements across application domains:

Task / Model	Baseline	Gaze-Regularized Variant	Absolute / Relative Gain
Egocentric VLM (Semantic Score)	0.6525	0.7316 / 0.7826	+12% (singular), +20% (aggreg.)
Robotic VLA (LIBERO avg, 30k)	82.0%	90.8%	+8.8%
Imitation learning (TCR, stat.)	40%	80%	+40% absolute
Gaze estimation (CLIP-Gaze)	8.65° (MAE)	6.97°	≈10–20% reduction
Segmentation (U-ResAtt + EFE)	89.59%	90.27%	+0.68 pp
Shape regularization ablation	90.27%	89.59% (w/o EFE)	Performance decline

• In VLMs for egocentric video, the gaze-regularized model shows 13% semantic score improvement for future event prediction and 7% for current activity understanding, outperforming baselines that do not leverage gaze (Pani et al., 24 Mar 2026, Pani et al., 24 Oct 2025).
• Gaze-regularized VLA models for robotics deliver 4–12% higher manipulation success rates, increased robustness to lighting/sensor perturbations, and more interpretable attention maps that strongly overlap with human fixation regions (Pani et al., 24 Mar 2026).
• Behavioral imitation with gaze regularization doubles task completion rate and halves collision rates, demonstrating that auxiliary gaze prediction regularizes spatial feature learning (Thakur et al., 2021).

6. Architectural Flexibility and Deployment Considerations

• Training-Time Only Regularization: Most frameworks require gaze input only at training; deployment relies solely on visual data, retaining zero-shot generalization capabilities. Pseudo-gaze models further enable gaze-style performance even when ground-truth eye tracking is unavailable at inference (Pani et al., 24 Oct 2025, Pani et al., 24 Mar 2026).
• Architectural Minimalism: Gaze-regularized attention can be introduced modularly—often as a lightweight block or regularizer—without major network redesign, facilitating integration with existing Transformer, CNN, and hybrid architectures (Pani et al., 24 Mar 2026, Pani et al., 24 Oct 2025, Yin et al., 2024).
• Hyperparameter Sensitivity: Regularization strength ($N$2) is a critical parameter. Overly strong regularization degrades performance (e.g., $N$3 for $N$4 vs. $N$5 for $N$6 in robotic VLA), while moderate values create an effective soft inductive bias (Pani et al., 24 Mar 2026).
• Interpretability: Gaze-regularized attention patterns provide post-hoc visualizations closely mirroring human fixations, improving the transparency and safety auditability of deployed systems (Pani et al., 24 Mar 2026, Wei et al., 25 Feb 2026).

7. Application Domains and Prospective Impact

• Egocentric Video and Scene Understanding: Gaze-regularized VLMs and Gaze-VLM advance both descriptive and predictive video understanding, with direct applications to assistive robotics, augmented reality, and human-machine interaction in real-world scenarios (Pani et al., 24 Mar 2026, Pani et al., 24 Oct 2025).
• Robotics and Skill Learning: Soft alignment with gaze distributions in VLA architectures accelerates robot learning, produces robust behaviors under challenging real-world variations, and enhances the interpretability of action selection (Pani et al., 24 Mar 2026).
• Gaze Estimation and HCI: Incorporating geometric shape priors, domain-aligned feature regularizers, and statistical weighting, gaze-regularized estimation frameworks achieve state-of-the-art cross-domain accuracy, essential for smartphone eye tracking, VR/AR, and generalized user intent modeling (Yin et al., 2024, Yang, 27 Nov 2025, Guo et al., 2019, Adebayo et al., 2024).
• Imitation and Behavioral Cloning: By enforcing human-attended feature encoding, gaze regularization yields policy networks with improved generalization to novel environments and tasks (Thakur et al., 2021).

Collectively, gaze-regularized training frameworks constitute a principled methodology for integrating rich human perceptual signals into deep learning architectures, enabling advances in accuracy, robustness, transparency, and generalization in visually-grounded artificial intelligence.