Gaze-Based Regularization in Imitation Learning

• The paper introduces GABRIL, a framework that aligns policy network attention with human gaze to mitigate causal confusion in imitation learning.
• The methodology integrates an auxiliary gaze alignment loss computed from dense gaze maps collected via calibrated eye trackers to guide feature extraction.
• Empirical results show significant improvements in performance, generalization, and sample efficiency across domains such as Atari games, autonomous driving, and robotics.

Gaze-Based Regularization in Imitation Learning (GABRIL) is a family of approaches for leveraging human gaze as an inductive bias to improve the robustness, generalization, and interpretability of deep visuomotor imitation learning. GABRIL specifically refers to the method introduced in "GAze-Based Regularization for Mitigating Causal Confusion in Imitation Learning" and related developments (Banayeeanzade et al., 25 Jul 2025), but is situated within a broader class of frameworks employing gaze-derived regularization losses to constrain policy learning toward human-attended, causally relevant visual features. This paradigm is motivated by the observation that human gaze during task execution reveals not only reactive attention but also embodies high-level intent and expert guidance for discerning critical state information—information that, when encoded in policy networks, can mitigate causal confusion and susceptibility to spurious correlations, while also conferring practical and theoretical advantages over standard behavioral cloning.

1. Theoretical Motivation and Problem Setup

Imitation learning (IL), and particularly behavior cloning (BC), involves optimizing a policy $\pi:\mathcal O \to \mathcal A$ to minimize the discrepancy between the agent's actions and expert-provided actions over demonstration data $(o_i, a_i)$: $$\mathcal L_{\rm BC}(\pi) = \frac{1}{N} \sum_{i=1}^N \ell(\pi(o_i), a_i)$$ where $\ell$ is typically a cross-entropy or regression loss (Banayeeanzade et al., 25 Jul 2025). A recurring failure mode in BC is "causal confusion": agents exploit spurious visual cues $t$ strongly correlated with $a_i$ in training data, rather than causal factors $s$. This leads to marked distribution shift failures—$P(\pi(o)\mid\mathrm{do}(s,t))$ may diverge across shifts in $t$ even when $s$ remains constant.

Human gaze, by contrast, is hypothesized (and empirically observed) to condition primarily on causal task-relevant factors and remains robust to confounders. Regularizing IL agents to align their internal feature saliency with human gaze thus provides an avenue for instilling causal invariance, both in practical policy learning and in principle, under certain identifiability assumptions on gaze generation (Banayeeanzade et al., 25 Jul 2025). Prior efforts in this direction (e.g., AGIL (Zhang et al., 2018), GMD (Liu et al., 2019, Chen et al., 2019), CGL (Saran et al., 2020)) varied in how gaze information was incorporated, with GABRIL unifying the approach as an auxiliary loss directly penalizing mismatch between network attention and demonstration gaze maps.

2. Gaze Data Collection, Processing, and Representation

Gaze for regularization is collected during expert demonstration using eye trackers (e.g., GP3 HD at 150 Hz, EyeLink-1000 at 1000 Hz), with calibration to ensure geometric correspondence between gaze coordinates and input images (Banayeeanzade et al., 25 Jul 2025, Saran et al., 2020). For each frame $(o_i, a_i)$0,

• Raw gaze coordinates $(o_i, a_i)$1 are recorded.
• To obtain a saliency mask $(o_i, a_i)$2, temporal smoothing is performed by accumulating gaze points over a window $(o_i, a_i)$3 with geometric decay (parameters $(o_i, a_i)$4, $(o_i, a_i)$5, spread $(o_i, a_i)$6), using a sum of Gaussians:

$(o_i, a_i)$7

The result is normalized to produce $(o_i, a_i)$8 per frame (Banayeeanzade et al., 25 Jul 2025).

Alternative representations exist: some methods use continuous gaze coordinates (Thakur et al., 2021), 2D heatmaps (Saran et al., 2020), patch-level gaze priors for transformer models (Pani et al., 24 Mar 2026), or gaze-centered 3D crops for manipulation in robotics (Takizawa et al., 25 Feb 2025). The key is that gaze regularization always operates with a saliency-like signal aligned to each observed state.

3. Gaze-Based Regularization: Losses and Network Integration

The canonical GABRIL framework (Banayeeanzade et al., 25 Jul 2025) introduces an auxiliary gaze alignment loss:

• Let $(o_i, a_i)$9 be the encoder’s final convolutional layer.
• A gaze predictor $$\mathcal L_{\rm BC}(\pi) = \frac{1}{N} \sum_{i=1}^N \ell(\pi(o_i), a_i)$$0, implemented as channel-mean-pooling, spatial softmax, and upsampling, generates a dense predicted attention map.
• The gaze-prediction loss is

$$\mathcal L_{\rm BC}(\pi) = \frac{1}{N} \sum_{i=1}^N \ell(\pi(o_i), a_i)$$1

• Full objective:

$$\mathcal L_{\rm BC}(\pi) = \frac{1}{N} \sum_{i=1}^N \ell(\pi(o_i), a_i)$$2

with $$\mathcal L_{\rm BC}(\pi) = \frac{1}{N} \sum_{i=1}^N \ell(\pi(o_i), a_i)$$3 controlling the strength of gaze regularization.

This approach has multiple correct instantiations:

• KL Regularization: Minimizing $$\mathcal L_{\rm BC}(\pi) = \frac{1}{N} \sum_{i=1}^N \ell(\pi(o_i), a_i)$$4 at patch level (Pani et al., 24 Mar 2026, Saran et al., 2020).
• Multi-Objective Prediction: Adding a gaze regression head and combining BC loss and MSE on predicted gaze (Thakur et al., 2021).
• Gaze-Modulated Dropout: Using gaze to modulate dropout rates spatially such that channels aligned with high-attention regions are less likely to be dropped during training (Chen et al., 2019, Liu et al., 2019).

All methods do not require gaze at test time. During training, gaze-based losses encourage the policy’s perceptual representations to place saliency on regions selected by human fixations, biasing feature extraction toward likely-causal content and away from distractors.

4. Empirical Evaluation and Impact

The effect of GABRIL and related regularizers has been established across domains:

• Atari Games: In 15 confounded games, GABRIL achieves mean “Advantage over BC” (ABC) +27.1% under confounders (vs. best prior +12.8%), and up to +179% gain over previous approaches (Banayeeanzade et al., 25 Jul 2025). Data efficiency is high; with as little as 20% of gaze data, ≈80% of maximal gain is retained.
• CARLA/Bench2Drive: GABRIL improves generalization to unseen routes: BC: 53.1% completion, GABRIL: 61.6% (+8.5% absolute, +76% relative to best baseline) (Banayeeanzade et al., 25 Jul 2025).
• Behavioral Cloning from Observation, T-REX: Coverage-based gaze losses yield average gains of +343% (BCO) and +390% (T-REX) in 20 Atari games (Saran et al., 2020).
• Robotic Manipulation: Gaze regularization (in both shallow and VLA transformers) improves sample efficiency and final task success on manipulation and bimanual tasks, with gains of 4–12% seen across several benchmarks and scenarios (Pani et al., 24 Mar 2026, Takizawa et al., 25 Feb 2025).
• Qualitative Effects: Saliency/visualization analyses consistently show GABRIL-trained policies place high attention on task-relevant cues, such as traffic signals (CARLA), the ball or paddle (Atari Breakout), or critical objects (robotic manipulation), while standard BC policies attend to distractors or uninformative regions (Banayeeanzade et al., 25 Jul 2025, Pani et al., 24 Mar 2026).

5. Comparison with Related Gaze Regularization Methods

GABRIL's explicit regularization differs from integration-based methods in both philosophy and implementation:

• Input Stacking or Gaze-Masked Augmentation: Early works (AGIL, GMD) added gaze-masked features as extra inputs or branches, requiring either gaze at test time or additional architecture (Zhang et al., 2018, Liu et al., 2019). GABRIL and CGL losses are strictly training-time, parameterless, and agnostic to downstream architecture.
• Dropout Modulation: Gaze-modulated dropout regularizes feature learning but does so as a feature-level mask; GABRIL acts directly on spatial activations, steering learned representations (Chen et al., 2019).
• Vision-Language-Action Models: Patch-level gaze priors regularize cross-attention maps, requiring only standard transformer mechanisms and no inference-time costs (Pani et al., 24 Mar 2026).
• Multi-Objective Heads: Some frameworks augment BC loss with gaze coordinate regression or auxiliary gaze heads (GRIL), but GABRIL achieves top performance with a single extra loss over shared encoders (Thakur et al., 2021).

In minimum-intervention regimes—where no test-time gaze or architectural growth is desired—GABRIL-aligned regularization and its variants are the principal techniques.

Method	Gaze Use	Parameter Increase	Test-Time Gaze	Empirical Gain (%)
GABRIL (Banayeeanzade et al., 25 Jul 2025)	Regularization	None	No	27–179
AGIL (Zhang et al., 2018)	Input/Branch	Yes	Yes	10–52
GMD [(Liu et al., 2019);1904...]	Dropout	None	No	1–29
VLA-KL (Pani et al., 24 Mar 2026)	KL Patch Loss	None	No	4–12

6. Interpretability, Robustness, and Limitations

A defining feature of GABRIL is the alignment of learned attention with human visual strategies. The regularization maps can be visualized directly to audit agent focus at test time, enhancing explainability and user trust (Banayeeanzade et al., 25 Jul 2025, Pani et al., 24 Mar 2026). Robustness under distribution shift improves: GABRIL policies are less prone to exploit confounders (shown by smaller performance drops under confounding augmentation), and show greater success under camera noise, lighting shifts, and cross-scene deployment (Pani et al., 24 Mar 2026).

Key limitations include:

• Gaze data collection is required at training, which may not always be feasible or high-quality (e.g., open-world robotics).
• Causal confusion from temporal correlations—“copycat” failures—are not fully addressed; gaze acts on instantaneous input, not history (Banayeeanzade et al., 25 Jul 2025).
• The regularization is effective when human gaze is well-correlated with the causal factors; multi-modal or ambiguous gaze may be less effective as a supervisory signal (Saran et al., 2020).
• Synthetic gaze (obtained via pretrained networks in the absence of real data) yields benefit but remains sensitive to the fidelity of the gaze predictor (Pani et al., 24 Mar 2026).

This suggests further integrating temporally-structured attention supervision or combining gaze with semantic/linguistic priors may expand GABRIL’s efficacy in complex long-horizon or compositional IL domains.

7. Practical Recommendations and Outlook

Empirical evidence across domains supports the recommendation that practitioners of IL collect gaze data during demonstration and employ gaze-based regularization—optimizing

$$\mathcal L_{\rm BC}(\pi) = \frac{1}{N} \sum_{i=1}^N \ell(\pi(o_i), a_i)$$5

with $$\mathcal L_{\rm BC}(\pi) = \frac{1}{N} \sum_{i=1}^N \ell(\pi(o_i), a_i)$$6 chosen by small-scale grid search (Banayeeanzade et al., 25 Jul 2025). Visualization and audit of alignment, as enabled by GABRIL’s gaze map predictions, are effective tools for both debugging and interpretability. GABRIL is compatible with, and frequently complements, other regularizers (dropout, contrastive) and is architecture-agnostic.

Ongoing research extends GABRIL principles to multi-modal domains (robotics, VLA models), investigates robustness under synthetic vs. real gaze priors, and explores the theoretical identification of causal variables via perceptual alignment. Expansion to online adaptation and temporal gaze aggregation represents a promising trajectory for future development.

References:

(Banayeeanzade et al., 25 Jul 2025): "GABRIL: Gaze-Based Regularization for Mitigating Causal Confusion in Imitation Learning" (Saran et al., 2020): "Efficiently Guiding Imitation Learning Agents with Human Gaze" (Pani et al., 24 Mar 2026): "Gaze-Regularized Vision-Language-Action Models for Robotic Manipulation" (Thakur et al., 2021): "Imitation Learning with Human Eye Gaze via Multi-Objective Prediction" (Takizawa et al., 25 Feb 2025): "Enhancing Reusability of Learned Skills for Robot Manipulation via Gaze and Bottleneck" (Chen et al., 2019, Liu et al., 2019, Zhang et al., 2018)