Gaze-Guided Contrastive Learning
- Gaze-guided contrastive learning is a paradigm that uses gaze signals to structure contrastive objectives, refining view selection and augmentation strategies.
- Methods like GazeCLR, ConGaze, and LG-Gaze demonstrate enhanced performance through subject-aware sampling, regression-aware loss design, and multimodal alignment.
- The approach extends beyond gaze estimation to applications in medical imaging and reinforcement learning, addressing challenges such as sparse, noisy gaze data.
Gaze-guided contrastive learning denotes a family of representation-learning methods in which gaze is not treated merely as a downstream regression target, but as a source of structure for defining invariances, selecting positives and negatives, weighting multimodal alignments, or supervising spatial attention. In the recent literature, the term spans several regimes: contrastive learning tailored to gaze estimation itself, multimodal gaze estimation via language-guided alignment, and systems in which expert gaze guides representation learning for diagnosis, retrieval, or control (Jindal et al., 2022, Du et al., 2023, Yin et al., 2024, Riju et al., 18 Aug 2025, Lee et al., 9 Oct 2025). A practical taxonomy, inferred from these works, is therefore broader than a single objective family.
1. Scope, terminology, and taxonomic boundaries
A recurring distinction in the literature separates gaze-aware contrastive learning from methods that merely inherit a contrastively pretrained backbone. In the stricter sense, gaze changes the contrastive problem itself: pair construction, augmentation policy, latent geometry, or loss weighting are designed around gaze semantics. In a broader sense, gaze may supervise another task’s contrastive learner, as in medical imaging or visual reinforcement learning. A third, weaker sense includes models that use CLIP-style priors for gaze estimation without introducing a new downstream contrastive objective. This suggests that the phrase “gaze-guided contrastive learning” is best read as an umbrella category rather than a single canonical method (Wang et al., 2023, Zhao et al., 2023, Liu et al., 27 Mar 2026).
| Paradigm | Role of gaze | Representative papers |
|---|---|---|
| Geometry- or subject-aware SSL for gaze estimation | Gaze shapes views, negatives, or equivariance | (Jindal et al., 2022, Du et al., 2022, Du et al., 2023) |
| Semi-supervised or regression-aware contrastive learning | Continuous gaze or gaze components determine similarity or ranking | (Maiti et al., 2023, Jiang et al., 2024, Tan et al., 26 May 2026) |
| Multimodal gaze estimation | Gaze is aligned with text or language-space prompts | (Yin et al., 2024, Wang et al., 2023) |
| Human-gaze-guided representation learning | Expert gaze supervises augmentation, pair mining, or attention | (Wang et al., 2023, Zhao et al., 2023, Riju et al., 18 Aug 2025, Liu et al., 27 Mar 2026, Lee et al., 9 Oct 2025) |
An early precursor outside image-based gaze estimation is CLRGaze, which adapts SimCLR to eye-movement time series by learning on velocity signals, 1-second segments at 500 Hz, a six-layer TCN encoder, and NT-Xent, reaching 84.6% accuracy on a mixed six-dataset biometric task and up to 97.3% on a single dataset (Bautista et al., 2020). This demonstrates that contrastive learning can be made gaze-specific even when gaze is the signal modality rather than the supervisory cue.
2. Geometry-aware and subject-aware self-supervision for gaze estimation
GazeCLR formalizes a central claim of the field: gaze estimation requires invariance to appearance changes that do not affect gaze, but equivariance to viewpoint changes because the same physical gaze vector changes coordinates across cameras in a predictable way. Its pretraining objective combines a single-view NT-Xent loss for gaze-preserving augmentations with a multi-view aligned NT-Xent loss after rotating embeddings into a common screen reference frame. In its notation, the overall loss is
Empirically, richer multi-view supervision improved within-dataset mean angular error from 7.72° with two views to 4.83° with four views, and single-subject batches were crucial: 4.83° with single-subject batches versus 23.58° with multi-subject batches. The method also improved cross-domain gaze estimation by as much as 17.2% relative improvement (Jindal et al., 2022).
FreeGaze and ConGaze attack a different failure mode: generic contrastive learning tends to organize representations around identity and appearance rather than gaze. FreeGaze retains a SimCLR-style objective,
$\mathcal{L}_{i,p,q} = -\log{\frac{\exp \Big(\text{sim}(z_i^p,z_i^q)/\tau\Big)}{\sum_{k=1}^N{\exp\Big(\text{sim}(z_i^p,z_k^q)/\tau\Big)}} ,$
but makes the training signal gaze-aware through subject-specific negative pair sampling and gaze-specific data augmentation, while operating on DCT inputs for efficiency. On ETH-XGaze, RGB-SimCLR yielded whereas FreeGaze reached ; on MPIIFaceGaze, RGB-SimCLR was and FreeGaze . The framework also reported 3.33× to 6.81× speedup in calibration and 1.59× to 1.67× in inference latency (Du et al., 2022).
ConGaze similarly replaces generic instance discrimination with gaze-aware structure. Its positive pairs are two gaze-preserving augmentations of the same face image, while negatives are different images of the same subject. Contrastive learning is applied in subject-specific embedding spaces through a subject-conditional projection module. The subject-specific loss is
$\mathcal{L}_{i, j} = -\log{\frac{\exp \Big(\text{sim}(\mathbf{z}_{i,j}^p,\mathbf{z}_{i,j}^q)/\tau\Big)}{\sum_{k=1}^{K_i}{\exp\Big(\text{sim}(\mathbf{z}_{i,j}^p,\mathbf{z}_{i,k}^q)/\tau\Big)}} .$
Across three public datasets, ConGaze outperformed existing unsupervised learning solutions by 6.7% to 22.5%, and in cross-dataset evaluations it improved over its supervised-learning-based counterpart by 15.1% to 24.6% (Du et al., 2023).
3. Regression-aware and semi-supervised contrastive formulations
A second branch of the literature modifies contrastive learning for the fact that gaze is a continuous regression problem. “Semi-supervised Contrastive Regression for Estimation of Eye Gaze” uses a two-stage SimCLR-derived pipeline: unlabeled ETH-XGaze images pretrain an encoder, then a regression head is fine-tuned with Huber loss. Its proposed contrastive loss combines a modified NT-Xent term with an off-diagonal cross-correlation penalty,
$\mathcal{L}=-\frac{1}{B}\sum_{i,j\in1}^{B} \log\frac{\exp(sim(p_i,p_j)/\tau)}{\sum_{k=1}^{2B}\mathbbm{1}_{k\neq i}\ \exp(sim(p_i,p_k)/\tau)} + \gamma \sum_{i}^{}\sum_{j\neq i}^{}C_{ij}^2 .$
On ETH-XGaze, the reported mean angular errors were 9.175° for SimCLR, 9.390° for Barlow Twins, 2.246° for SimCLR with Deeplab Encoder, 2.152° for the full method, and 3.212° for a reduced-parameter variant. Loss ablations gave 4.606° for NT-Xent, 4.3361° for Barlow Twins loss, and 3.212° for the proposed loss (Maiti et al., 2023).
DSCL extends this regression-aware line by arguing that a single scalar ranking is ill-posed for multi-target gaze labels such as pitch and yaw. It therefore uses Jacobian regularization to disentangle the latent space into component-specific subspaces,
$\mathcal{L}_{\pmb{J} = \sum_{m=1}^M \sum_{z=1}^{M} \sum_{n=1}^{N} \mathbbm{1}_{m \neq z} |\pmb{J}_{m,n} \cdot \pmb{J}_{z,n}|,$
constructs a hard binary mask from the regressor Jacobian, and then performs subspace-specific ranking-based contrastive learning. Its total objective is
$\mathcal{L}_{Total} = \mathcal{L}_{reg}(Y,\hat Y) + \gamma \mathcal{L}_{\pmb{J} + w_{SC} \mathcal{L}_{SC} + w_{UC} \mathcal{L}_{UC} + w_{UR}\mathcal{L}_{UR}.$
The paper reports competitive performance using only 20%, 10%, and 5% of the annotated data under both in-domain and cross-domain settings. On Gaze360, a baseline at 5% labels gave 23.00°, whereas baseline + DSCL gave 14.21° (Tan et al., 26 May 2026).
A more unconventional formulation is EM-IB, which rejects standard instance discrimination as a poor fit for full-face gaze pretraining and instead uses eye-masked reconstruction plus a reconstruction-based contrastive term:
$\mathcal{L}_{i,p,q} = -\log{\frac{\exp \Big(\text{sim}(z_i^p,z_i^q)/\tau\Big)}{\sum_{k=1}^N{\exp\Big(\text{sim}(z_i^p,z_k^q)/\tau\Big)}} ,$0
with
$\mathcal{L}_{i,p,q} = -\log{\frac{\exp \Big(\text{sim}(z_i^p,z_i^q)/\tau\Big)}{\sum_{k=1}^N{\exp\Big(\text{sim}(z_i^p,z_k^q)/\tau\Big)}} ,$1
Because the contrast is defined by eye/gaze relevance rather than image identity, the method is better understood as region-aware, reconstruction-based gaze-guided self-supervision (Jiang et al., 2024).
4. Multimodal and language-guided gaze estimation
Vision-language methods introduce a different notion of guidance: text or language embeddings regularize gaze representations. GazeCLIP is the weaker form. It uses pretrained CLIP, a template prompt of the form “A photo of a face gazing [class]”, cross-attention fusion, and standard L1 or L2 regression losses on yaw and pitch. The paper is explicit that it does not introduce a new downstream contrastive loss; the contrastive component is inherited from CLIP pretraining only. This makes GazeCLIP relevant to the topic, but not a full example of gaze-guided contrastive learning in the strict objective-design sense (Wang et al., 2023).
LG-Gaze is the stronger form. It reframes gaze estimation as a vision-language alignment problem, uses continuous learned prompts, constructs gaze embeddings by geometry-aware interpolation over anchor embeddings, and introduces a multimodal contrastive regression loss in which negative terms are weighted by gaze-label relation:
$\mathcal{L}_{i,p,q} = -\log{\frac{\exp \Big(\text{sim}(z_i^p,z_i^q)/\tau\Big)}{\sum_{k=1}^N{\exp\Big(\text{sim}(z_i^p,z_k^q)/\tau\Big)}} ,$2
with bidirectional loss
$\mathcal{L}_{i,p,q} = -\log{\frac{\exp \Big(\text{sim}(z_i^p,z_i^q)/\tau\Big)}{\sum_{k=1}^N{\exp\Big(\text{sim}(z_i^p,z_k^q)/\tau\Big)}} ,$3
The full objective is
$\mathcal{L}_{i,p,q} = -\log{\frac{\exp \Big(\text{sim}(z_i^p,z_i^q)/\tau\Big)}{\sum_{k=1}^N{\exp\Big(\text{sim}(z_i^p,z_k^q)/\tau\Big)}} ,$4
Across four cross-domain evaluation tasks, the main domain-generalization comparison reported averages of 8.57 for the CNN baseline, 6.97 for CLIP-Gaze, and 6.84 for LG-Gaze. Loss ablations gave 8.65 for $\mathcal{L}_{i,p,q} = -\log{\frac{\exp \Big(\text{sim}(z_i^p,z_i^q)/\tau\Big)}{\sum_{k=1}^N{\exp\Big(\text{sim}(z_i^p,z_k^q)/\tau\Big)}} ,$5 only, 7.01 for $\mathcal{L}_{i,p,q} = -\log{\frac{\exp \Big(\text{sim}(z_i^p,z_i^q)/\tau\Big)}{\sum_{k=1}^N{\exp\Big(\text{sim}(z_i^p,z_k^q)/\tau\Big)}} ,$6, and 6.84 for $\mathcal{L}_{i,p,q} = -\log{\frac{\exp \Big(\text{sim}(z_i^p,z_i^q)/\tau\Big)}{\sum_{k=1}^N{\exp\Big(\text{sim}(z_i^p,z_k^q)/\tau\Big)}} ,$7 (Yin et al., 2024).
These results support an important conceptual distinction. Some language-guided methods are language-guided in modality but not gaze-aware in contrastive structure, whereas LG-Gaze is both: the language branch supplies the semantic scaffold, and gaze geometry changes how negatives are treated. This suggests that “guidance” can enter either through the choice of modality or through the construction of the contrastive objective itself.
5. Human gaze as a supervisory signal for contrastive learning
A broader literature uses human gaze to guide contrastive learning for tasks other than gaze estimation. FocusContrast is the clearest example at the augmentation level. It trains a U-Net to predict radiologist gaze maps from knee X-rays and then uses the predicted attention to constrain view generation for SimCLR, MoCo, and BYOL. Focus cutout accepts a view only when gaze-region IOU exceeds 0.9; focus crop uses 0.8; focus mask suppresses the 80% less informative areas. The module improved classification accuracy by 4.0~7.0% on the knee X-ray dataset (Wang et al., 2023).
McGIP uses gaze differently: not to preserve attended pixels in a view, but to mine cross-image positive pairs. If two medical images elicit similar radiologist gaze, they are treated as semantically similar in contrastive learning. The method builds a gaze affinity matrix $\mathcal{L}_{i,p,q} = -\log{\frac{\exp \Big(\text{sim}(z_i^p,z_i^q)/\tau\Big)}{\sum_{k=1}^N{\exp\Big(\text{sim}(z_i^p,z_k^q)/\tau\Big)}} ,$8 and uses pairs with $\mathcal{L}_{i,p,q} = -\log{\frac{\exp \Big(\text{sim}(z_i^p,z_i^q)/\tau\Big)}{\sum_{k=1}^N{\exp\Big(\text{sim}(z_i^p,z_k^q)/\tau\Big)}} ,$9 as additional weighted positives inside a generic contrastive framework. On INbreast and Tufts dental panoramic X-rays, McGIP improved MoCo, BYOL, and SimSiam across multiple backbones, and often outperformed supervised same-label pair mining (Zhao et al., 2023).
Chest X-ray systems extend this pattern from pair mining to attention supervision. “Eyes on the Image” introduces multimodal contrastive alignment among image, text, and gaze embeddings together with a composite gaze-attention loss combining MSE, KL divergence, Pearson-correlation loss, and center-of-mass alignment. The abstract reports that incorporating fixations improves F1 score from 0.597 to 0.631 (+5.70%) and AUC from 0.821 to 0.849 (+3.41%); all reported test results use image-only inference, so gaze functions as privileged training-time supervision (Riju et al., 18 Aug 2025). CoGaze pushes further into vision-language pretraining: global image-report alignment is handled by hybrid-positive contrastive learning, disease semantics by cross-modal classification, and gaze by a soft Jensen–Shannon-divergence loss between transcript-to-patch similarity distributions and gaze-derived patch distributions. The paper reports up to +2.0% CheXbertF1, +1.2% BLEU2, +23.2% AUROC, and +12.2% Precision@1, while using gaze only during pretraining (Liu et al., 27 Mar 2026).
The most radical extrapolation appears in visual reinforcement learning. “Gaze on the Prize” adds a learnable Gaussian foveal attention module and trains it with return-guided triplets built from visually similar states with different returns. Its contrastive component is a triplet loss on attended representations,
0
with 1. The reported outcome is up to 2.4x improvement in sample efficiency on ManiSkill3 manipulation tasks (Lee et al., 9 Oct 2025). In this context, “gaze-guided” refers not to human eye tracking but to a learned foveal mechanism whose attention is shaped by contrastive supervision.
6. Empirical regularities, misconceptions, and unresolved issues
Several empirical regularities recur across this literature. First, naive contrastive learning is often a weak baseline for gaze-sensitive problems. FreeGaze reports RGB-SimCLR at 2 on ETH-XGaze and DCT-SimCLR at 3, versus 4 for FreeGaze (Du et al., 2022). GazeCLR reports that single-subject batches outperform multi-subject batches by a very large margin (Jindal et al., 2022). Semi-supervised contrastive regression reports 9.175° for SimCLR and 2.152° for its full method (Maiti et al., 2023). These findings support a general conclusion: unless pair construction and invariances are explicitly aligned with gaze semantics, contrastive learning tends to encode the wrong factors.
Second, architecture and inductive bias matter nearly as much as the loss. In semi-supervised contrastive regression, a SimCLR-with-Deeplab-Encoder baseline already reaches 2.246°, close to the full method’s 2.152° (Maiti et al., 2023). In ConGaze, the shift from ConEye to the subject-conditional projector gives a substantial reduction from 16.3 to 9.9 on ETH-XGaze and from 12.2 to 5.2 on E-C (Du et al., 2023). This suggests that contrastive objectives alone do not determine performance; gaze-specific augmentations, projection spaces, and subspace decompositions are equally consequential.
Third, the phrase “gaze-guided” is used inconsistently. GazeCLIP is closely related because it leverages CLIP’s pretrained contrastive space, yet it does not perform gaze-specific contrastive learning during downstream training (Wang et al., 2023). By contrast, LG-Gaze changes negative weighting, CoGaze uses gaze as a probabilistic prior over transcript-patch alignments, and McGIP uses gaze to mine cross-image positives (Yin et al., 2024, Liu et al., 27 Mar 2026, Zhao et al., 2023). A common misconception is therefore to equate any multimodal gaze model with gaze-guided contrastive learning. The stricter reading requires gaze to alter the representation-learning objective or the sampling mechanism.
Finally, unresolved issues are explicit in the papers. ConGaze requires subject identity during pretraining (Du et al., 2023). GazeCLR requires synchronized calibrated multi-view data and camera geometry (Jindal et al., 2022). Semi-supervised contrastive regression leaves many hyperparameters unspecified and restricts experiments to ETH-XGaze (Maiti et al., 2023). LG-Gaze contains an apparent inconsistency between the verbal description of negative weighting and the formula 5, and its interpolation depends on a fixed anchor grid (Yin et al., 2024). Medical approaches repeatedly note that gaze is sparse, noisy, and available for only a small fraction of pretraining data (Zhao et al., 2023, Liu et al., 27 Mar 2026). This suggests that future work will likely focus on more principled continuous-label contrastive objectives, weaker dependence on subject metadata or calibrated rigs, and better use of limited gaze supervision as a probabilistic rather than binary prior.