---
title: Gaze-Guided Contrastive Learning
url: https://www.emergentmind.com/topics/gaze-guided-contrastive-learning
type: topic
---

# Gaze-Guided Contrastive Learning

Gaze-guided contrastive learning denotes a family of representation-learning methods in which gaze is not treated merely as a downstream regression target, but as a source of structure for defining invariances, selecting positives and negatives, weighting multimodal alignments, or supervising spatial attention. In the recent literature, the term spans several regimes: contrastive learning tailored to gaze estimation itself, multimodal gaze estimation via language-guided alignment, and systems in which expert gaze guides representation learning for diagnosis, retrieval, or control [2210.13404][2309.04506][2411.08606][2508.13068][2510.08442]. A practical taxonomy, inferred from these works, is therefore broader than a single objective family.

## 1. Scope, terminology, and taxonomic boundaries

A recurring distinction in the literature separates **gaze-aware contrastive learning** from methods that merely inherit a contrastively pretrained backbone. In the stricter sense, gaze changes the contrastive problem itself: pair construction, augmentation policy, latent geometry, or loss weighting are designed around gaze semantics. In a broader sense, gaze may supervise another task’s contrastive learner, as in medical imaging or visual reinforcement learning. A third, weaker sense includes models that use CLIP-style priors for gaze estimation without introducing a new downstream contrastive objective. This suggests that the phrase “gaze-guided contrastive learning” is best read as an umbrella category rather than a single canonical method [2401.00260][2312.06069][2603.26049].

| Paradigm | Role of gaze | Representative papers |
|---|---|---|
| Geometry- or subject-aware SSL for gaze estimation | Gaze shapes views, negatives, or equivariance | [2210.13404], [2209.06692], [2309.04506] |
| Semi-supervised or regression-aware contrastive learning | Continuous gaze or gaze components determine similarity or ranking | [2308.02784], [2407.00315], [2605.27080] |
| Multimodal gaze estimation | Gaze is aligned with text or language-space prompts | [2411.08606], [2401.00260] |
| Human-gaze-guided representation learning | Expert gaze supervises augmentation, pair mining, or attention | [2305.08826], [2312.06069], [2508.13068], [2603.26049], [2510.08442] |

An early precursor outside image-based gaze estimation is CLRGaze, which adapts SimCLR to eye-movement time series by learning on velocity signals, 1-second segments at 500 Hz, a six-layer TCN encoder, and NT-Xent, reaching **84.6%** accuracy on a mixed six-dataset biometric task and up to **97.3%** on a single dataset [2010.13046]. This demonstrates that contrastive learning can be made gaze-specific even when gaze is the signal modality rather than the supervisory cue.

## 2. Geometry-aware and subject-aware self-supervision for gaze estimation

GazeCLR formalizes a central claim of the field: gaze estimation requires **invariance** to appearance changes that do not affect gaze, but **equivariance** to viewpoint changes because the same physical gaze vector changes coordinates across cameras in a predictable way. Its pretraining objective combines a single-view NT-Xent loss for gaze-preserving augmentations with a multi-view aligned NT-Xent loss after rotating embeddings into a common screen reference frame. In its notation, the overall loss is
$$
\mathscr{L}^{O} = \dfrac{1}{2B} \sum_{i=1}^{|V|} \sum_{b=1}^{B} \biggl( \mathscr{L}^{I}(z^b_{v_i}, z'^b_{v_i}) + \mathscr{L}^{I}(z'^b_{v_i}, z^b_{v_i}) + \sum_{j=1, j \neq i}^{|V|} \mathscr{L}^{E}(\bar{z}^b_{v_i}, \bar{z}^b_{v_j}) \biggr).
$$
Empirically, richer multi-view supervision improved within-dataset mean angular error from **7.72°** with two views to **4.83°** with four views, and single-subject batches were crucial: **4.83°** with single-subject batches versus **23.58°** with multi-subject batches. The method also improved cross-domain gaze estimation by as much as **17.2% relative improvement** [2210.13404].

FreeGaze and ConGaze attack a different failure mode: generic contrastive learning tends to organize representations around identity and appearance rather than gaze. FreeGaze retains a SimCLR-style objective,
$$
\mathcal{L}_{i,p,q} = -\log{\frac{\exp \Big(\text{sim}(z_i^p,z_i^q)/\tau\Big)}{\sum_{k=1}^N{\exp\Big(\text{sim}(z_i^p,z_k^q)/\tau\Big)}} ,
$$
but makes the training signal gaze-aware through **subject-specific negative pair sampling** and **gaze-specific data augmentation**, while operating on DCT inputs for efficiency. On ETH-XGaze, **RGB-SimCLR** yielded **\(8.38^\circ \pm 1.63^\circ\)** whereas **FreeGaze** reached **\(2.95^\circ \pm 0.51^\circ\)**; on MPIIFaceGaze, **RGB-SimCLR** was **\(9.65^\circ \pm 1.12^\circ\)** and **FreeGaze** **\(5.07^\circ \pm 1.03^\circ\)**. The framework also reported **3.33× to 6.81×** speedup in calibration and **1.59× to 1.67×** in inference latency [2209.06692].

ConGaze similarly replaces generic instance discrimination with gaze-aware structure. Its positive pairs are two gaze-preserving augmentations of the same face image, while negatives are different images of the **same subject**. Contrastive learning is applied in **subject-specific embedding spaces** through a subject-conditional projection module. The subject-specific loss is
$$
\mathcal{L}_{i, j} = -\log{\frac{\exp \Big(\text{sim}(\mathbf{z}_{i,j}^p,\mathbf{z}_{i,j}^q)/\tau\Big)}{\sum_{k=1}^{K_i}{\exp\Big(\text{sim}(\mathbf{z}_{i,j}^p,\mathbf{z}_{i,k}^q)/\tau\Big)}} .
$$
Across three public datasets, ConGaze outperformed existing unsupervised learning solutions by **6.7% to 22.5%**, and in cross-dataset evaluations it improved over its supervised-learning-based counterpart by **15.1% to 24.6%** [2309.04506].

## 3. Regression-aware and semi-supervised contrastive formulations

A second branch of the literature modifies contrastive learning for the fact that gaze is a **continuous regression problem**. “Semi-supervised Contrastive Regression for Estimation of Eye Gaze” uses a two-stage SimCLR-derived pipeline: unlabeled ETH-XGaze images pretrain an encoder, then a regression head is fine-tuned with Huber loss. Its proposed contrastive loss combines a modified NT-Xent term with an off-diagonal cross-correlation penalty,
$$
\mathcal{L}=-\frac{1}{B}\sum_{i,j\in1}^{B} \log\frac{\exp(sim(p_i,p_j)/\tau)}{\sum_{k=1}^{2B}\mathbbm{1}_{k\neq i}\ \exp(sim(p_i,p_k)/\tau)} + \gamma \sum_{i}^{}\sum_{j\neq i}^{}C_{ij}^2 .
$$
On ETH-XGaze, the reported mean angular errors were **9.175°** for SimCLR, **9.390°** for Barlow Twins, **2.246°** for SimCLR with Deeplab Encoder, **2.152°** for the full method, and **3.212°** for a reduced-parameter variant. Loss ablations gave **4.606°** for NT-Xent, **4.3361°** for Barlow Twins loss, and **3.212°** for the proposed loss [2308.02784].

DSCL extends this regression-aware line by arguing that a single scalar ranking is ill-posed for multi-target gaze labels such as pitch and yaw. It therefore uses **Jacobian regularization** to disentangle the latent space into component-specific subspaces,
$$
\mathcal{L}_{\pmb{J} = \sum_{m=1}^M \sum_{z=1}^{M} \sum_{n=1}^{N} \mathbbm{1}_{m \neq z} |\pmb{J}_{m,n} \cdot \pmb{J}_{z,n}|,
$$
constructs a hard binary mask from the regressor Jacobian, and then performs subspace-specific ranking-based contrastive learning. Its total objective is
$$
\mathcal{L}_{Total} = \mathcal{L}_{reg}(Y,\hat Y) + \gamma \mathcal{L}_{\pmb{J} + w_{SC} \mathcal{L}_{SC}
 + w_{UC} \mathcal{L}_{UC} + w_{UR}\mathcal{L}_{UR}.
$$
The paper reports competitive performance using only **20%**, **10%**, and **5%** of the annotated data under both in-domain and cross-domain settings. On Gaze360, a baseline at **5%** labels gave **23.00°**, whereas baseline + DSCL gave **14.21°** [2605.27080].

A more unconventional formulation is EM-IB, which rejects standard instance discrimination as a poor fit for full-face gaze pretraining and instead uses eye-masked reconstruction plus a reconstruction-based contrastive term:
$$
\mathcal{L}_{contr} = \max\big( |\mathcal{F}(\mathbf{f}^{+},\mathbf{f}^{0})- \mathbf{I}_e^{gt}| - |\mathcal{F}(\mathbf{f}^{-},\mathbf{f}^{0})- \mathbf{I}_e^{gt}|, 0 \big),
$$
with
$$
\mathcal{L}=\mathcal{L}_{rec} + \lambda\mathcal{L}_{contr}, \qquad \lambda = 0.01 .
$$
Because the contrast is defined by **eye/gaze relevance** rather than image identity, the method is better understood as region-aware, reconstruction-based gaze-guided self-supervision [2407.00315].

## 4. Multimodal and language-guided gaze estimation

Vision-language methods introduce a different notion of guidance: text or language embeddings regularize gaze representations. GazeCLIP is the weaker form. It uses pretrained CLIP, a template prompt of the form “A photo of a face gazing [class]”, cross-attention fusion, and standard L1 or L2 regression losses on yaw and pitch. The paper is explicit that it **does not** introduce a new downstream contrastive loss; the contrastive component is inherited from CLIP pretraining only. This makes GazeCLIP relevant to the topic, but not a full example of gaze-guided contrastive learning in the strict objective-design sense [2401.00260].

LG-Gaze is the stronger form. It reframes gaze estimation as a **vision-language alignment problem**, uses continuous learned prompts, constructs gaze embeddings by geometry-aware interpolation over anchor embeddings, and introduces a multimodal contrastive regression loss in which negative terms are weighted by gaze-label relation:
$$
w(i,j)=\cos(\boldsymbol{g}_i,\boldsymbol{g}_j),
$$
with bidirectional loss
$$
\ell_{MCR}=\ell_{i 2 t}+\ell_{t 2 i}.
$$
The full objective is
$$
\mathcal{L} = \lambda_{1} \ell_{Geo}  + \lambda_{2}\ell_{MCR}  + \lambda_{3}\ell_{Gaze},
\qquad
\lambda_{1} = \lambda_{2} = \lambda_{3} = 1.0 .
$$
Across four cross-domain evaluation tasks, the main domain-generalization comparison reported averages of **8.57** for the CNN baseline, **6.97** for CLIP-Gaze, and **6.84** for LG-Gaze. Loss ablations gave **8.65** for $\ell_{Gaze}$ only, **7.01** for $\ell_{MCR} + \ell_{Gaze}$, and **6.84** for $\ell_{Geo} + \ell_{MCR} + \ell_{Gaze}$ [2411.08606].

These results support an important conceptual distinction. Some language-guided methods are **language-guided in modality but not gaze-aware in contrastive structure**, whereas LG-Gaze is both: the language branch supplies the semantic scaffold, and gaze geometry changes how negatives are treated. This suggests that “guidance” can enter either through the choice of modality or through the construction of the contrastive objective itself.

## 5. Human gaze as a supervisory signal for contrastive learning

A broader literature uses **human gaze** to guide contrastive learning for tasks other than gaze estimation. FocusContrast is the clearest example at the augmentation level. It trains a U-Net to predict radiologist gaze maps from knee X-rays and then uses the predicted attention to constrain view generation for SimCLR, MoCo, and BYOL. Focus cutout accepts a view only when gaze-region IOU exceeds **0.9**; focus crop uses **0.8**; focus mask suppresses the **80% less informative areas**. The module improved classification accuracy by **4.0~7.0%** on the knee X-ray dataset [2305.08826].

McGIP uses gaze differently: not to preserve attended pixels in a view, but to mine **cross-image positive pairs**. If two medical images elicit similar radiologist gaze, they are treated as semantically similar in contrastive learning. The method builds a gaze affinity matrix \(A_{ij}\) and uses pairs with \(A_{ij}\ge t\) as additional weighted positives inside a generic contrastive framework. On INbreast and Tufts dental panoramic X-rays, McGIP improved MoCo, BYOL, and SimSiam across multiple backbones, and often outperformed supervised same-label pair mining [2312.06069].

Chest X-ray systems extend this pattern from pair mining to attention supervision. “Eyes on the Image” introduces multimodal contrastive alignment among image, text, and gaze embeddings together with a composite gaze-attention loss combining MSE, KL divergence, Pearson-correlation loss, and center-of-mass alignment. The abstract reports that incorporating fixations improves **F1 score from 0.597 to 0.631 (+5.70%)** and **AUC from 0.821 to 0.849 (+3.41%)**; all reported test results use **image-only inference**, so gaze functions as privileged training-time supervision [2508.13068]. CoGaze pushes further into vision-language pretraining: global image-report alignment is handled by **hybrid-positive contrastive learning**, disease semantics by cross-modal classification, and gaze by a soft Jensen–Shannon-divergence loss between transcript-to-patch similarity distributions and gaze-derived patch distributions. The paper reports up to **+2.0% CheXbertF1**, **+1.2% BLEU2**, **+23.2% AUROC**, and **+12.2% Precision@1**, while using gaze only during pretraining [2603.26049].

The most radical extrapolation appears in visual reinforcement learning. “Gaze on the Prize” adds a learnable Gaussian foveal attention module and trains it with return-guided triplets built from visually similar states with different returns. Its contrastive component is a triplet loss on attended representations,
$$
\mathcal{L}_{\text{con}(\theta)} = \mathbb{E}_{(o_a, o_+, o_-)} \left[ \max\bigl(0, D(z_a, z_+) - D(z_a, z_-) + \alpha \bigr) \right],
$$
with \(\alpha=0.5\). The reported outcome is up to **2.4x improvement in sample efficiency** on ManiSkill3 manipulation tasks [2510.08442]. In this context, “gaze-guided” refers not to human eye tracking but to a learned foveal mechanism whose attention is shaped by contrastive supervision.

## 6. Empirical regularities, misconceptions, and unresolved issues

Several empirical regularities recur across this literature. First, **naive contrastive learning is often a weak baseline for gaze-sensitive problems**. FreeGaze reports **RGB-SimCLR** at **\(8.38^\circ \pm 1.63^\circ\)** on ETH-XGaze and **DCT-SimCLR** at **\(10.11^\circ \pm 2.00^\circ\)**, versus **\(2.95^\circ \pm 0.51^\circ\)** for FreeGaze [2209.06692]. GazeCLR reports that single-subject batches outperform multi-subject batches by a very large margin [2210.13404]. Semi-supervised contrastive regression reports **9.175°** for SimCLR and **2.152°** for its full method [2308.02784]. These findings support a general conclusion: unless pair construction and invariances are explicitly aligned with gaze semantics, contrastive learning tends to encode the wrong factors.

Second, **architecture and inductive bias matter nearly as much as the loss**. In semi-supervised contrastive regression, a SimCLR-with-Deeplab-Encoder baseline already reaches **2.246°**, close to the full method’s **2.152°** [2308.02784]. In ConGaze, the shift from ConEye to the subject-conditional projector gives a substantial reduction from **16.3** to **9.9** on ETH-XGaze and from **12.2** to **5.2** on E-C [2309.04506]. This suggests that contrastive objectives alone do not determine performance; gaze-specific augmentations, projection spaces, and subspace decompositions are equally consequential.

Third, the phrase **“gaze-guided” is used inconsistently**. GazeCLIP is closely related because it leverages CLIP’s pretrained contrastive space, yet it does **not** perform gaze-specific contrastive learning during downstream training [2401.00260]. By contrast, LG-Gaze changes negative weighting, CoGaze uses gaze as a probabilistic prior over transcript-patch alignments, and McGIP uses gaze to mine cross-image positives [2411.08606][2603.26049][2312.06069]. A common misconception is therefore to equate any multimodal gaze model with gaze-guided contrastive learning. The stricter reading requires gaze to alter the representation-learning objective or the sampling mechanism.

Finally, unresolved issues are explicit in the papers. ConGaze requires **subject identity during pretraining** [2309.04506]. GazeCLR requires **synchronized calibrated multi-view data** and camera geometry [2210.13404]. Semi-supervised contrastive regression leaves many hyperparameters unspecified and restricts experiments to ETH-XGaze [2308.02784]. LG-Gaze contains an apparent inconsistency between the verbal description of negative weighting and the formula \(w(i,j)=\cos(\boldsymbol{g}_i,\boldsymbol{g}_j)\), and its interpolation depends on a fixed anchor grid [2411.08606]. Medical approaches repeatedly note that gaze is **sparse**, **noisy**, and available for only a small fraction of pretraining data [2312.06069][2603.26049]. This suggests that future work will likely focus on more principled continuous-label contrastive objectives, weaker dependence on subject metadata or calibrated rigs, and better use of limited gaze supervision as a probabilistic rather than binary prior.

Source: https://www.emergentmind.com/topics/gaze-guided-contrastive-learning