Gaze-Regularized Attention Mechanism

• Gaze-Regularized Attention Mechanism is defined as neural techniques that incorporate human gaze cues to guide attention allocation during training.
• It employs gaze-derived signals such as fixation heatmaps to reweight features and adjust attention scores in tasks like image captioning and surgical recognition.
• Empirical results indicate improved performance and more interpretable models, as evidenced by enhanced metrics in medical imaging, activity recognition, and robotic manipulation.

Gaze-regularized attention mechanism denotes a family of neural attention formulations in which human gaze, fixation maps, or gaze-derived latent variables are used to shape where a model allocates representational capacity. Across image captioning, video captioning, activity recognition, medical imaging, imitation learning, egocentric vision-language modeling, and robotic manipulation, the common objective is to bias internal attention toward task-relevant regions that humans actually inspect, rather than relying solely on weak supervision from class labels or text tokens (Sugano et al., 2016, Yu et al., 2017, Awale et al., 2022, Zhu et al., 2022, Pani et al., 24 Oct 2025, Pani et al., 24 Mar 2026). In the narrow sense, gaze regularization uses human or synthetic human-like attention during training to constrain model attention; in a broader contemporary sense, the term also covers gaze-inspired attention control mechanisms that emulate selective looking without direct human gaze supervision.

1. Historical emergence and problem setting

Early work established that human gaze can complement learned neural attention even in scene-centric tasks. “Seeing with Humans: Gaze-Assisted Neural Image Captioning” introduced a split-attention captioner in which human fixation weights modulate two attention streams, one for fixated regions and one for non-fixated regions, showing that gaze can complement machine attention for semantic scene understanding (Sugano et al., 2016). “Supervising Neural Attention Models for Video Captioning by Human Gaze Data” extended this logic to video, using a recurrent gaze predictor to generate frame-level spatial attention for caption generation and reporting gains both on automatic language metrics and human judgment (Yu et al., 2017).

Subsequent work moved from captioning to recognition and control. In egocentric activity recognition, gaze was modeled as a structured discrete latent variable with variational training and direct optimization over discrete samples, explicitly addressing fixation noise, uncertain gaze–attention correlation, and the absence of gaze at test time (Min et al., 2020). In imitation learning, GRIL treated gaze prediction as an auxiliary objective rather than an explicit attention mask, so the shared encoder learned features predictive of both action and gaze (Thakur et al., 2021). In surgical activity recognition, human gaze supervised a spatio-temporal attention module embedded in I3D, and the study stated that it was, to the authors’ knowledge, the first to use human gaze for surgical activity recognition (Awale et al., 2022). In chest X-ray classification, GG-CAM directly regulated CAM formation using radiologist gaze and emphasized that the extension is lightweight, with $3$ additional trainable parameters for regulating the learning process (Zhu et al., 2022).

Recent work generalizes gaze regularization to transformer-based VLMs and VLA models. In egocentric VLMs, patch-level attention distributions are aligned with gaze-derived patch distributions through KL divergence, while gaze is used only during training (Pani et al., 24 Oct 2025). In robotic manipulation, temporally aggregated gaze heatmaps are converted into patch-level distributions and aligned with transformer attention with no architectural modifications or inference-time overhead (Pani et al., 24 Mar 2026). This trajectory shows a shift from gaze as a direct input signal toward gaze as a training-time inductive bias.

2. Gaze representation and signal construction

A gaze-regularized mechanism begins with a representation of human attention. The most common construction starts from fixation coordinates and converts them into a dense heatmap. In surgical video recognition, a pretrained saliency model outputs a single fixation point per frame $(x_t,y_t)$, which is converted into a normalized Gaussian heatmap

$$G_t(i,j)=\exp\!\left(-\frac{\|[i,j]-[x_t,y_t]\|^2}{2\sigma^2}\right),$$

with $\sum_{i,j}G_t(i,j)=1$, and stacked over time to form $G\in\mathbb{R}^{T\times H\times W}$ (Awale et al., 2022). Video captioning uses a related construction: a binary fixation map per frame is smoothed with a Gaussian of $\sigma=2.0$ and $\ell_1$-normalized to a probability distribution $G^\tau\in\Delta_{49\times49}$ (Yu et al., 2017).

Transformer-based formulations typically require patch-level rather than pixel-level supervision. In egocentric VLMs, a gaze point $g_t=(g_t^x,g_t^y)$ is first converted into a spatial heatmap

$m_t=\pi\bigl(G_\sigma * \mathbf{1}(g_t)\bigr),$

then temporally aggregated over a short window with optical-flow warping and occlusion filtering,

$(x_t,y_t)$0

and finally projected to a patch distribution $(x_t,y_t)$1 by summing over patch support (Pani et al., 24 Oct 2025). The robotic manipulation setting uses an analogous procedure: per-frame heatmaps $(x_t,y_t)$2 are aggregated with weights $(x_t,y_t)$3, partitioned into non-overlapping patches $(x_t,y_t)$4, and normalized into a patch-level probability vector $(x_t,y_t)$5 (Pani et al., 24 Mar 2026).

Not all formulations treat gaze as a deterministic heatmap. In egocentric activity recognition, gaze is represented as a structured discrete latent variable

$(x_t,y_t)$6

with a variational approximation $(x_t,y_t)$7 learned from intermediate I3D features. A sampled $(x_t,y_t)$8 is then mapped to a smooth attention tensor through

$(x_t,y_t)$9

This formulation explicitly models uncertainty due to blinks, saccades, and dissociation between fixation and task relevance (Min et al., 2020).

3. Architectural forms of gaze regularization

The simplest architectural pattern is multiplicative reweighting of intermediate visual features. In surgical activity recognition, an I3D-based model computes a soft spatio-temporal attention map $$G_t(i,j)=\exp\!\left(-\frac{\|[i,j]-[x_t,y_t]\|^2}{2\sigma^2}\right),$$0 from an intermediate feature tensor $$G_t(i,j)=\exp\!\left(-\frac{\|[i,j]-[x_t,y_t]\|^2}{2\sigma^2}\right),$$1 and applies it as

$$G_t(i,j)=\exp\!\left(-\frac{\|[i,j]-[x_t,y_t]\|^2}{2\sigma^2}\right),$$2

The reweighted volume is then passed to the remaining I3D layers and classification head (Awale et al., 2022). A closely related pattern appears in egocentric activity recognition, where the attended feature volume is

$$G_t(i,j)=\exp\!\left(-\frac{\|[i,j]-[x_t,y_t]\|^2}{2\sigma^2}\right),$$3

In both cases, gaze supervision affects recognition indirectly by altering the internal feature field (Min et al., 2020).

A second pattern uses gaze to modify the attention-score function itself. In gaze-assisted image captioning, the split-attention module defines

$$G_t(i,j)=\exp\!\left(-\frac{\|[i,j]-[x_t,y_t]\|^2}{2\sigma^2}\right),$$4

followed by softmax normalization over spatial locations. When $$G_t(i,j)=\exp\!\left(-\frac{\|[i,j]-[x_t,y_t]\|^2}{2\sigma^2}\right),$$5, the model routes the region through $$G_t(i,j)=\exp\!\left(-\frac{\|[i,j]-[x_t,y_t]\|^2}{2\sigma^2}\right),$$6; when $$G_t(i,j)=\exp\!\left(-\frac{\|[i,j]-[x_t,y_t]\|^2}{2\sigma^2}\right),$$7, it routes through $$G_t(i,j)=\exp\!\left(-\frac{\|[i,j]-[x_t,y_t]\|^2}{2\sigma^2}\right),$$8 (Sugano et al., 2016). This design does not force attention to equal gaze; rather, gaze acts as a gated bias over the scoring function.

A third pattern regularizes class activation maps. GG-CAM replaces a standard global-average-pooling-plus-linear head with an explicit CAM layer,

$$G_t(i,j)=\exp\!\left(-\frac{\|[i,j]-[x_t,y_t]\|^2}{2\sigma^2}\right),$$9

and supervises only the true-class slice $\sum_{i,j}G_t(i,j)=1$0 against a normalized human-gaze heat map $\sum_{i,j}G_t(i,j)=1$1 via a selective MSE after sigmoid scaling (Zhu et al., 2022). Here gaze regularization is placed directly on the model’s explanatory map, rather than on a latent attention map.

Transformer-era mechanisms usually operate at the level of queries, keys, or explicit attention distributions. One egocentric VLM modifies each attention head by adding a projected gaze query,

$\sum_{i,j}G_t(i,j)=1$2

and defines the regularizer as either MSE or KL between attention and gaze (Pani et al., 24 Mar 2026). Voila-A injects gaze into the key stream of a Perceiver-style resampler through

$\sum_{i,j}G_t(i,j)=1$3

where $\sum_{i,j}G_t(i,j)=1$4, thereby biasing latent-to-image attention without an explicit auxiliary gaze loss (Yan et al., 2023). In gaze-regularized VLA and egocentric VLM settings, the model extracts an attention distribution over visual patches from a cross-attention layer and matches it directly to a patch-level gaze distribution with KL divergence (Pani et al., 24 Mar 2026, Pani et al., 24 Oct 2025).

4. Objective functions and optimization regimes

A canonical gaze-regularized objective augments the task loss with a discrepancy term between model attention and gaze. In surgical recognition, the classification loss

$\sum_{i,j}G_t(i,j)=1$5

is combined with either a pixel-wise cross-entropy or KL divergence gaze loss,

$\sum_{i,j}G_t(i,j)=1$6

with $\sum_{i,j}G_t(i,j)=1$7 chosen by cross-validation and typical values $\sum_{i,j}G_t(i,j)=1$8–$\sum_{i,j}G_t(i,j)=1$9 (Awale et al., 2022). In transformer-based egocentric understanding, the total objective is

$G\in\mathbb{R}^{T\times H\times W}$0

with

$G\in\mathbb{R}^{T\times H\times W}$1

and the reported best setting is $G\in\mathbb{R}^{T\times H\times W}$2 (Pani et al., 24 Oct 2025). In robotic manipulation, the same form appears as

$G\in\mathbb{R}^{T\times H\times W}$3

with grid search reporting $G\in\mathbb{R}^{T\times H\times W}$4 as the best final-performance setting, $G\in\mathbb{R}^{T\times H\times W}$5 still beneficial, and $G\in\mathbb{R}^{T\times H\times W}$6 harmful because it over-constrains the model (Pani et al., 24 Mar 2026).

Some methods learn the balance between task and gaze terms. GG-CAM follows Kendall et al. (2018) and optimizes

$G\in\mathbb{R}^{T\times H\times W}$7

where $G\in\mathbb{R}^{T\times H\times W}$8 and $G\in\mathbb{R}^{T\times H\times W}$9 are trainable uncertainty scalars, initialized so that gaze supervision dominates early training (Zhu et al., 2022). This makes gaze regularization part of a multitask uncertainty-weighting scheme rather than a fixed penalty.

Other models use gaze without an explicit gaze-consistency loss. In split-attention image captioning, there is no separate “gaze consistency loss” or $\sigma=2.0$0 penalty on $\sigma=2.0$1 versus $\sigma=2.0$2; gaze enters only through the gated form of $\sigma=2.0$3 (Sugano et al., 2016). In GEAN, the recurrent gaze predictor is trained first with frame-wise cross-entropy on gaze maps, then frozen, and the captioning stage adds no gaze term to the caption loss because spatial attention is driven entirely by predicted gaze (Yu et al., 2017). In GRIL, the mechanism is even looser: the gaze head predicts gaze coordinates with an MSE loss, and the action head shares the encoder, so gaze supervision regularizes attention implicitly through multi-objective representation learning rather than through an explicit attention map (Thakur et al., 2021).

A persistent design distinction concerns inference. Several systems explicitly state that gaze is training-only: GG-CAM-modified CNNs do not require human attention as an input when fully trained (Zhu et al., 2022); the gaze-regularized VLA framework has zero inference overhead (Pani et al., 24 Mar 2026); and gaze-regularized egocentric VLMs revert to standard attention at inference (Pani et al., 24 Oct 2025). Other systems learn a gaze model so that ground-truth gaze is unnecessary at test time, as in recurrent gaze prediction for video captioning and variational gaze prediction for egocentric activity recognition (Yu et al., 2017, Min et al., 2020).

5. Reported empirical behavior across domains

The empirical record consistently associates gaze regularization with improved discrimination, cleaner localization, and more human-aligned explanations.

Domain and paper	Reported change
Surgical activity recognition (Awale et al., 2022)	I3D without attention: 77.6 accuracy, 70.1 F1, 72.8 edit; I3D + human-gaze guided attention: 85.4 ± 5.3 accuracy, 80.6 F1, 81.2 edit
Chest X-ray classification (Zhu et al., 2022)	ResNet50 median AUC: 0.721 → 0.776; EfficientNetV2-s median AUC: 0.723 → 0.801
Image captioning (Sugano et al., 2016)	Baseline CIDEr 0.626 → split-attention 0.638; BLEU-4 0.237 → 0.245
Egocentric activity recognition (Min et al., 2020)	EGTEA split 1: 59.5/67.5 → 62.8/69.6 for mean-class accuracy / overall accuracy
Egocentric VLM future prediction (Pani et al., 24 Oct 2025)	OpenFlamingo semantic score 0.6525 → 0.7505; OpenFlamingo activity understanding 0.7176 → 0.7848
Robotic manipulation VLA (Pani et al., 24 Mar 2026)	LIBERO-Spatial final success 85.9% → 95.5%; OpenVLA backbone +5.7 pts overall

Qualitative findings are similarly consistent. In surgery, gaze supervision is reported to reduce reliance on background or distracting textures and to force attention onto the needle, tools, and tissue interface (Awale et al., 2022). In chest X-rays, GG-CAM heat maps align more closely with radiologist gaze foci, and localization improves for pneumonia and cardiomegaly tasks (Zhu et al., 2022). In image captioning, the split-attention model is better at discovering small but important objects such as “kite,” “knife,” and “fire hydrant,” while retaining the flexibility to attend to non-fixated but semantically critical regions through the $\sigma=2.0$4 branch (Sugano et al., 2016). In robotic manipulation, baseline attention is described as diffuse, whereas gaze-regularized attention concentrates on manipulated objects, target bins, and plate edges; the Top-10 overlap metric is reported as baseline $\sigma=2.0$5 versus regularized $\sigma=2.0$6 (Pani et al., 24 Mar 2026). In egocentric VLMs, Top-10 patch overlap rises from $\sigma=2.0$7 to $\sigma=2.0$8, and human evaluation reports a hallucination rate $\sigma=2.0$9 drop from $\ell_1$0 to $\ell_1$1 (Pani et al., 24 Oct 2025).

A plausible implication is that gaze regularization is most effective when the task depends on small spatial supports, weak label supervision, or anticipatory cues that are not reliably recoverable from labels alone.

6. Interpretability, limitations, and adjacent mechanisms

The main interpretability claim is not merely that gaze improves accuracy, but that it regulates where the model grounds its prediction. GG-CAM states that the method brings better interpretability and facilitates weakly supervised pathology localization and analysis (Zhu et al., 2022). Robotic manipulation work argues that human-aligned attention patterns mirror human strategies and enhance trust in robotic systems (Pani et al., 24 Mar 2026). Egocentric VLM results connect gaze alignment with reduced visual hallucinations and more grounded generations (Pani et al., 24 Oct 2025).

Several limitations recur. The egocentric activity literature notes measurement errors due to blinking and rapid eye movements, uncertain correlation between gaze and attention, and the fact that gaze data is often unavailable in real-world situations (Min et al., 2020). Video captioning identifies the expense of eye-tracking data collection, the fact that gaze supervises only spatial attention while temporal attention remains implicit, and the need to tune the uniform prior $\ell_1$2 to avoid over-focusing (Yu et al., 2017). Egocentric VLM work reports that calibrated gaze datasets remain scarce, downsampling to $\ell_1$3 fps may miss fast events, and optical-flow-based occlusion filtering can discard valid gaze or include noisy points (Pani et al., 24 Oct 2025). Robotic manipulation shows that overly large regularization weights can degrade performance, indicating that gaze should bias attention rather than replace task supervision (Pani et al., 24 Mar 2026).

A common misconception is that gaze-regularized attention always requires gaze at inference. Many prominent formulations explicitly avoid that requirement through training-only regularization, learned gaze predictors, or latent gaze models (Zhu et al., 2022, Yu et al., 2017, Min et al., 2020, Pani et al., 24 Oct 2025, Pani et al., 24 Mar 2026). Another misconception is that gaze regularization is identical to generic saliency supervision. In image captioning, replacing gaze with bottom-up saliency yields smaller gains, and the study argues that human gaze contains complementary semantic signals (Sugano et al., 2016).

Recent literature also broadens the term “gaze” beyond direct human supervision. “Learning to See What You Need: Gaze Attention for Multimodal LLMs” groups visual tokens into fixed-size gaze regions and dynamically selects the top-$\ell_1$4 regions per decoding step, reducing visual KV use by up to $\ell_1$5 without auxiliary human-gaze supervision (Song et al., 13 May 2026). “GazeVLM” introduces internally generated <LOOK> tokens and a continuous suppression bias over cross-attention logits, again without human gaze labels at deployment (Ebouky et al., 8 May 2026). “Conscious Gaze” is training-free and inference-time, using Harsanyi interaction variance to detect when visual grounding is needed and then reorienting mid-layer attention toward visual tokens (Bu et al., 5 Dec 2025). These mechanisms are best understood as gaze-inspired attention control rather than gaze regularization in the strict human-supervision sense.

In contemporary usage, therefore, gaze-regularized attention mechanism names both a specific supervised paradigm—aligning model attention with human gaze distributions—and a broader design philosophy: constraining attention to behave more like selective looking, whether the supervisory source is real gaze, synthetic gaze, a learned gaze prior, or an internally generated focusing policy.