---
title: Gaze-Regularized Attention Mechanism
url: https://www.emergentmind.com/topics/gaze-regularized-attention-mechanism
type: topic
---

# Gaze-Regularized Attention Mechanism

Gaze-regularized attention mechanism denotes a family of neural attention formulations in which human gaze, fixation maps, or gaze-derived latent variables are used to shape where a model allocates representational capacity. Across image captioning, video captioning, activity recognition, medical imaging, imitation learning, egocentric vision-language modeling, and robotic manipulation, the common objective is to bias internal attention toward task-relevant regions that humans actually inspect, rather than relying solely on weak supervision from class labels or text tokens [1608.05203] [1707.06029] [2203.04752] [2202.07107] [2510.21356] [2603.23202]. In the narrow sense, gaze regularization uses human or synthetic human-like attention during training to constrain model attention; in a broader contemporary sense, the term also covers gaze-inspired attention control mechanisms that emulate selective looking without direct human gaze supervision.

## 1. Historical emergence and problem setting

Early work established that human gaze can complement learned neural attention even in scene-centric tasks. “Seeing with Humans: Gaze-Assisted Neural Image Captioning” introduced a split-attention captioner in which human fixation weights modulate two attention streams, one for fixated regions and one for non-fixated regions, showing that gaze can complement machine attention for semantic scene understanding [1608.05203]. “Supervising Neural Attention Models for Video Captioning by Human Gaze Data” extended this logic to video, using a recurrent gaze predictor to generate frame-level spatial attention for caption generation and reporting gains both on automatic language metrics and human judgment [1707.06029].

Subsequent work moved from captioning to recognition and control. In egocentric activity recognition, gaze was modeled as a structured discrete latent variable with variational training and direct optimization over discrete samples, explicitly addressing fixation noise, uncertain gaze–attention correlation, and the absence of gaze at test time [2011.03920]. In imitation learning, GRIL treated gaze prediction as an auxiliary objective rather than an explicit attention mask, so the shared encoder learned features predictive of both action and gaze [2102.13008]. In surgical activity recognition, human gaze supervised a spatio-temporal attention module embedded in I3D, and the study stated that it was, to the authors’ knowledge, the first to use human gaze for surgical activity recognition [2203.04752]. In chest X-ray classification, GG-CAM directly regulated CAM formation using radiologist gaze and emphasized that the extension is lightweight, with $3$ additional trainable parameters for regulating the learning process [2202.07107].

Recent work generalizes gaze regularization to transformer-based VLMs and VLA models. In egocentric VLMs, patch-level attention distributions are aligned with gaze-derived patch distributions through KL divergence, while gaze is used only during training [2510.21356]. In robotic manipulation, temporally aggregated gaze heatmaps are converted into patch-level distributions and aligned with transformer attention with no architectural modifications or inference-time overhead [2603.23202]. This trajectory shows a shift from gaze as a direct input signal toward gaze as a training-time inductive bias.

## 2. Gaze representation and signal construction

A gaze-regularized mechanism begins with a representation of human attention. The most common construction starts from fixation coordinates and converts them into a dense heatmap. In surgical video recognition, a pretrained saliency model outputs a single fixation point per frame $(x_t,y_t)$, which is converted into a normalized Gaussian heatmap
$$
G_t(i,j)=\exp\!\left(-\frac{\|[i,j]-[x_t,y_t]\|^2}{2\sigma^2}\right),
$$
with $\sum_{i,j}G_t(i,j)=1$, and stacked over time to form $G\in\mathbb{R}^{T\times H\times W}$ [2203.04752]. Video captioning uses a related construction: a binary fixation map per frame is smoothed with a Gaussian of $\sigma=2.0$ and $\ell_1$-normalized to a probability distribution $G^\tau\in\Delta_{49\times49}$ [1707.06029].

Transformer-based formulations typically require patch-level rather than pixel-level supervision. In egocentric VLMs, a gaze point $g_t=(g_t^x,g_t^y)$ is first converted into a spatial heatmap
$$
m_t=\pi\bigl(G_\sigma * \mathbf{1}(g_t)\bigr),
$$
then temporally aggregated over a short window with optical-flow warping and occlusion filtering,
$$
H_t=\pi\!\Bigl(\sum_{\tau=t-\delta}^{t} o_{\tau}\,\bigl(f_{\tau\to t}\!\circ\,m_{\tau}\bigr)\Bigr),
$$
and finally projected to a patch distribution $\tilde H_{t,i}$ by summing over patch support [2510.21356]. The robotic manipulation setting uses an analogous procedure: per-frame heatmaps $H_{t+\delta}^i$ are aggregated with weights $w_\delta$, partitioned into non-overlapping patches $\{\mathcal P_j\}$, and normalized into a patch-level probability vector $G_t^i$ [2603.23202].

Not all formulations treat gaze as a deterministic heatmap. In egocentric activity recognition, gaze is represented as a structured discrete latent variable
$$
z\in\mathcal Z=\{1,\dots,T\}\times\{1,\dots,H'\}\times\{1,\dots,W'\},
$$
with a variational approximation $q_\phi(z\mid x)$ learned from intermediate I3D features. A sampled $z^*$ is then mapped to a smooth attention tensor through
$$
A(z^*)=\sigma\bigl(W\,\onehot(z^*)\bigr)\in[0,1]^{T\times H'\times W'}.
$$
This formulation explicitly models uncertainty due to blinks, saccades, and dissociation between fixation and task relevance [2011.03920].

## 3. Architectural forms of gaze regularization

The simplest architectural pattern is multiplicative reweighting of intermediate visual features. In surgical activity recognition, an I3D-based model computes a soft spatio-temporal attention map $A\in[0,1]^{T\times H'\times W'}$ from an intermediate feature tensor $X\in\mathbb{R}^{C\times T\times H'\times W'}$ and applies it as
$$
X'=(1+A)\odot X.
$$
The reweighted volume is then passed to the remaining I3D layers and classification head [2203.04752]. A closely related pattern appears in egocentric activity recognition, where the attended feature volume is
$$
F_{\rm att}=F+A(z^*)\odot F.
$$
In both cases, gaze supervision affects recognition indirectly by altering the internal feature field [2011.03920].

A second pattern uses gaze to modify the attention-score function itself. In gaze-assisted image captioning, the split-attention module defines
$$
e_{t,i}=g_i\,w_{\rm pos}^T p_{t,i}+(1-g_i)\,w_{\rm neg}^T p_{t,i}+c_{\rm att},
$$
followed by softmax normalization over spatial locations. When $g_i\approx 1$, the model routes the region through $w_{\rm pos}$; when $g_i\approx 0$, it routes through $w_{\rm neg}$ [1608.05203]. This design does not force attention to equal gaze; rather, gaze acts as a gated bias over the scoring function.

A third pattern regularizes class activation maps. GG-CAM replaces a standard global-average-pooling-plus-linear head with an explicit CAM layer,
$$
\Omega_{c,i,j}=\sum_{k=1}^G \Lambda_{c,k}\,A_{k,i,j},\qquad
y_c=\frac{1}{HW}\sum_{i,j}\Omega_{c,i,j}+\lambda_c,
$$
and supervises only the true-class slice $\Omega^Y$ against a normalized human-gaze heat map $\Psi$ via a selective MSE after sigmoid scaling [2202.07107]. Here gaze regularization is placed directly on the model’s explanatory map, rather than on a latent attention map.

Transformer-era mechanisms usually operate at the level of queries, keys, or explicit attention distributions. One egocentric VLM modifies each attention head by adding a projected gaze query,
$$
Q'_i = Q_{t,i} + W_g\cdot \mathrm{vec}(G),
$$
and defines the regularizer as either MSE or KL between attention and gaze [2603.23190]. Voila-A injects gaze into the key stream of a Perceiver-style resampler through
$$
K = K_{\rm img}+K_{\rm gaze},
$$
where $K_{\rm gaze}=W_k^g[G;P]$, thereby biasing latent-to-image attention without an explicit auxiliary gaze loss [2401.09454]. In gaze-regularized VLA and egocentric VLM settings, the model extracts an attention distribution over visual patches from a cross-attention layer and matches it directly to a patch-level gaze distribution with KL divergence [2603.23202] [2510.21356].

## 4. Objective functions and optimization regimes

A canonical gaze-regularized objective augments the task loss with a discrepancy term between model attention and gaze. In surgical recognition, the classification loss
$$
L_{\rm class}=-\sum_{k=1}^K y_k\log p_k
$$
is combined with either a pixel-wise cross-entropy or KL divergence gaze loss,
$$
L = L_{\rm class}+\lambda L_{\rm gaze},
$$
with $\lambda>0$ chosen by cross-validation and typical values $0.1$–$1.0$ [2203.04752]. In transformer-based egocentric understanding, the total objective is
$$
\mathcal L_{\rm total}=\mathcal L_{\rm task}+\lambda\,\mathcal L_{\rm gaze},
$$
with
$$
\mathcal L_{\rm gaze}=\sum_t D_{KL}(A_t\Vert \tilde H_t),
$$
and the reported best setting is $\lambda\approx 100$ [2510.21356]. In robotic manipulation, the same form appears as
$$
\mathcal L_{\rm total}=L_{\rm task}+\lambda\,D_{KL}(G_t^i\|S_t^i),
$$
with grid search reporting $\lambda=0.001$ as the best final-performance setting, $\lambda=0.01$ still beneficial, and $\lambda\ge 1$ harmful because it over-constrains the model [2603.23202].

Some methods learn the balance between task and gaze terms. GG-CAM follows Kendall et al. (2018) and optimizes
$$
\mathcal L=
\frac{1}{2\sigma_{sm}^2}\mathcal L_{sm}
+\frac{1}{\sigma_{ce}^2}\mathcal L_{ce}
+\ln(\sigma_{sm}+1)+\ln(\sigma_{ce}+1),
$$
where $\sigma_{sm}$ and $\sigma_{ce}$ are trainable uncertainty scalars, initialized so that gaze supervision dominates early training [2202.07107]. This makes gaze regularization part of a multitask uncertainty-weighting scheme rather than a fixed penalty.

Other models use gaze without an explicit gaze-consistency loss. In split-attention image captioning, there is no separate “gaze consistency loss” or $L_2$ penalty on $\alpha$ versus $g$; gaze enters only through the gated form of $e_{t,i}$ [1608.05203]. In GEAN, the recurrent gaze predictor is trained first with frame-wise cross-entropy on gaze maps, then frozen, and the captioning stage adds no gaze term to the caption loss because spatial attention is driven entirely by predicted gaze [1707.06029]. In GRIL, the mechanism is even looser: the gaze head predicts gaze coordinates with an MSE loss, and the action head shares the encoder, so gaze supervision regularizes attention implicitly through multi-objective representation learning rather than through an explicit attention map [2102.13008].

A persistent design distinction concerns inference. Several systems explicitly state that gaze is training-only: GG-CAM-modified CNNs do not require human attention as an input when fully trained [2202.07107]; the gaze-regularized VLA framework has zero inference overhead [2603.23202]; and gaze-regularized egocentric VLMs revert to standard attention at inference [2510.21356]. Other systems learn a gaze model so that ground-truth gaze is unnecessary at test time, as in recurrent gaze prediction for video captioning and variational gaze prediction for egocentric activity recognition [1707.06029] [2011.03920].

## 5. Reported empirical behavior across domains

The empirical record consistently associates gaze regularization with improved discrimination, cleaner localization, and more human-aligned explanations.

| Domain and paper | Reported change |
|---|---|
| Surgical activity recognition [2203.04752] | I3D without attention: 77.6 accuracy, 70.1 F1, 72.8 edit; I3D + human-gaze guided attention: 85.4 ± 5.3 accuracy, 80.6 F1, 81.2 edit |
| Chest X-ray classification [2202.07107] | ResNet50 median AUC: 0.721 → 0.776; EfficientNetV2-s median AUC: 0.723 → 0.801 |
| Image captioning [1608.05203] | Baseline CIDEr 0.626 → split-attention 0.638; BLEU-4 0.237 → 0.245 |
| Egocentric activity recognition [2011.03920] | EGTEA split 1: 59.5/67.5 → 62.8/69.6 for mean-class accuracy / overall accuracy |
| Egocentric VLM future prediction [2510.21356] | OpenFlamingo semantic score 0.6525 → 0.7505; OpenFlamingo activity understanding 0.7176 → 0.7848 |
| Robotic manipulation VLA [2603.23202] | LIBERO-Spatial final success 85.9% → 95.5%; OpenVLA backbone +5.7 pts overall |

Qualitative findings are similarly consistent. In surgery, gaze supervision is reported to reduce reliance on background or distracting textures and to force attention onto the needle, tools, and tissue interface [2203.04752]. In chest X-rays, GG-CAM heat maps align more closely with radiologist gaze foci, and localization improves for pneumonia and cardiomegaly tasks [2202.07107]. In image captioning, the split-attention model is better at discovering small but important objects such as “kite,” “knife,” and “fire hydrant,” while retaining the flexibility to attend to non-fixated but semantically critical regions through the $w_{\rm neg}$ branch [1608.05203]. In robotic manipulation, baseline attention is described as diffuse, whereas gaze-regularized attention concentrates on manipulated objects, target bins, and plate edges; the Top-10 overlap metric is reported as baseline $\sim 19\%$ versus regularized $\sim 51\%$ [2603.23202]. In egocentric VLMs, Top-10 patch overlap rises from $42\%$ to $68\%$, and human evaluation reports a hallucination rate $C_I$ drop from $0.205$ to $0.140$ [2510.21356].

A plausible implication is that gaze regularization is most effective when the task depends on small spatial supports, weak label supervision, or anticipatory cues that are not reliably recoverable from labels alone.

## 6. Interpretability, limitations, and adjacent mechanisms

The main interpretability claim is not merely that gaze improves accuracy, but that it regulates *where* the model grounds its prediction. GG-CAM states that the method brings better interpretability and facilitates weakly supervised pathology localization and analysis [2202.07107]. Robotic manipulation work argues that human-aligned attention patterns mirror human strategies and enhance trust in robotic systems [2603.23202]. Egocentric VLM results connect gaze alignment with reduced visual hallucinations and more grounded generations [2510.21356].

Several limitations recur. The egocentric activity literature notes measurement errors due to blinking and rapid eye movements, uncertain correlation between gaze and attention, and the fact that gaze data is often unavailable in real-world situations [2011.03920]. Video captioning identifies the expense of eye-tracking data collection, the fact that gaze supervises only spatial attention while temporal attention remains implicit, and the need to tune the uniform prior $\lambda$ to avoid over-focusing [1707.06029]. Egocentric VLM work reports that calibrated gaze datasets remain scarce, downsampling to $1$ fps may miss fast events, and optical-flow-based occlusion filtering can discard valid gaze or include noisy points [2510.21356]. Robotic manipulation shows that overly large regularization weights can degrade performance, indicating that gaze should bias attention rather than replace task supervision [2603.23202].

A common misconception is that gaze-regularized attention always requires gaze at inference. Many prominent formulations explicitly avoid that requirement through training-only regularization, learned gaze predictors, or latent gaze models [2202.07107] [1707.06029] [2011.03920] [2510.21356] [2603.23202]. Another misconception is that gaze regularization is identical to generic saliency supervision. In image captioning, replacing gaze with bottom-up saliency yields smaller gains, and the study argues that human gaze contains complementary semantic signals [1608.05203].

Recent literature also broadens the term “gaze” beyond direct human supervision. “Learning to See What You Need: Gaze Attention for Multimodal Large Language Models” groups visual tokens into fixed-size gaze regions and dynamically selects the top-$K$ regions per decoding step, reducing visual KV use by up to $90\%$ without auxiliary human-gaze supervision [2605.13080]. “GazeVLM” introduces internally generated `<LOOK>` tokens and a continuous suppression bias over cross-attention logits, again without human gaze labels at deployment [2605.07817]. “Conscious Gaze” is training-free and inference-time, using Harsanyi interaction variance to detect when visual grounding is needed and then reorienting mid-layer attention toward visual tokens [2512.05546]. These mechanisms are best understood as gaze-inspired attention control rather than gaze regularization in the strict human-supervision sense.

In contemporary usage, therefore, gaze-regularized attention mechanism names both a specific supervised paradigm—aligning model attention with human gaze distributions—and a broader design philosophy: constraining attention to behave more like selective looking, whether the supervisory source is real gaze, synthetic gaze, a learned gaze prior, or an internally generated focusing policy.

Source: https://www.emergentmind.com/topics/gaze-regularized-attention-mechanism