Investigating Bias and Fairness in Appearance-based Gaze Estimation

Abstract: While appearance-based gaze estimation has achieved significant improvements in accuracy and domain adaptation, the fairness of these systems across different demographic groups remains largely unexplored. To date, there is no comprehensive benchmark quantifying algorithmic bias in gaze estimation. This paper presents the first extensive evaluation of fairness in appearance-based gaze estimation, focusing on ethnicity and gender attributes. We establish a fairness baseline by analyzing state-of-the-art models using standard fairness metrics, revealing significant performance disparities. Furthermore, we evaluate the effectiveness of existing bias mitigation strategies when applied to the gaze domain and show that their fairness contributions are limited. We summarize key insights and open issues. Overall, our work calls for research into developing robust, equitable gaze estimators. To support future research and reproducibility, we publicly release our annotations, code, and trained models at: github.com/akgulburak/gaze-estimation-fairness

• The paper establishes the first fairness benchmark for appearance-based gaze estimation by augmenting datasets with ethnicity and gender annotations.
• It quantifies bias in state-of-the-art models using metrics like Wasserstein and Kolmogorov–Smirnov distances, revealing significant disparities across groups.
• Mitigation strategies such as oversampling, resampling, and loss reweighting show inconsistent improvements, highlighting the need for innovative bias reduction methods.

Investigating Bias and Fairness in Appearance-based Gaze Estimation

Introduction

Despite the rapid advancement of appearance-based gaze estimation methods and their broad adoption across real-world domains, the systematic exploration of bias and fairness in these models has been conspicuously lacking. This paper addresses pivotal gaps by delivering the first comprehensive benchmark for fairness in appearance-based gaze estimation, centered on the sensitive attributes of ethnicity and gender. The authors extend current datasets with demographic annotations, quantify bias in SOTA gaze estimation models, and scrutinize the impact of canonical bias mitigation strategies.

Figure 1: Overview of the three main contributions—dataset annotation with sensitive attributes, quantitative fairness analysis, and benchmarking bias mitigation methods for gaze estimation.

Demographic Annotation Pipeline

The absence of demographic attributes in standard gaze datasets constitutes a primary hurdle for fairness research in this domain. The authors augment the Gaze360 and GazeCapture datasets with both ethnicity and gender labels using an automated annotation pipeline that relies on the FairFace model, followed by human validation. Subjects with ambiguous or unclassifiable demographics are labeled as ‘Unsure’ and excluded from subsequent analyses. Three ethnicity labels (Caucasian, Asian, Afro-American) and two gender labels (male, female) ensure sufficient granularity for robust comparative evaluation.

Figure 3: Automated annotation pipeline for ethnicity and gender using FairFace, with human-in-the-loop disambiguation.

Performance and Fairness Metrics

Gaze estimation is treated as a multivariate regression problem over angular gaze coordinates. Performance is measured by mean angular error, computed separately for each demographic group. To quantify fairness, the authors employ three regression-specific disparity metrics:

• Wasserstein Distance (WS): Quantifies mismatch in error distributions between groups.
• Kolmogorov–Smirnov Distance (KS): Captures the supremum difference in cumulative error distributions.
• Two-sample t-test (t, p-values): Tests statistical significance of error distribution disparity.

A model is considered fully fair if $WS = 0$, $KS = 0$, and $p > 0.05$.

Baseline Bias Analysis in Gaze Datasets and Models

The Gaze360 and GazeCapture datasets exhibit pronounced representation bias, especially across ethnicities. For example, Gaze360 contains nearly 10× fewer Afro-American samples compared to Caucasian, while the gender distribution is balanced. This underlying data imbalance manifests as severe prediction bias in all evaluated SOTA models (CrossGaze, MCGaze, L2CS-Net, PureGaze, GazeTR), as evidenced by consistently elevated WS, KS, and statistically significant t-values ($p \ll 0.05$) for most ethnicity comparisons, particularly Caucasian–Afro-American.

Notably, gender-based disparities are less pronounced, which is traced back to more balanced data and possibly associated with lower variance in facial features for gaze regression. However, residual unfairness remains observable.

Bias Mitigation Pipelines and Efficacy

Figure 2: System-level pipeline showing dataset annotation, model training, and bias mitigation/calibration for fairness analysis.

Classic bias mitigation techniques—oversampling, resampling (reweighting), and in-training loss reweighting—are systematically benchmarked. Preprocessing approaches aim to equalize group sample cardinality, while in-processing approaches upweight minority group loss contributions.

Key Results

• Oversampling and resampling yield modest and sometimes inconsistent improvements in fairness across models and sensitive attributes. For instance, resampling provides the highest number of fairness improvements, but the effect size is far from closing the fairness gap, and in many cases, it fails to achieve statistical parity.
• Loss reweighting shows slightly better consistency but is still insufficient to robustly mitigate bias across sensitive attributes or across datasets.
• In certain circumstances, these mitigation strategies degrade both fairness and accuracy, highlighting fundamental limitations of naïve rebalancing.
• For several model–dataset–group combinations, even after mitigation, WS and KS values remain far from zero, and the t-test indicates persistent statistically significant disparity.

These findings empirically validate that appearance-based gaze estimation exposes similar, if not more severe, fairness vulnerabilities as seen in other facial analytic tasks. Additionally, counterintuitively, data balance at the sampling level (e.g., equal male–female split) does not guarantee group fairness in predictions, corroborating insights from broader work on intersectional fairness [cheong2023towards, wang2019balanced].

Practical and Theoretical Implications

The outcomes underscore several crucial points for the field:

• Appearance-based gaze models encode and amplify dataset-level demographic imbalances, translating into disparate prediction quality across groups—most notably along ethnicity axes.
• Current mainstream fairness rectification strategies such as sampling- and loss-based reweighting are not robustly effective for the complex, highly entangled visual signals underpinning gaze regression. This suggests that bias in appearance-based regression arises not merely from label imbalance, but from deeper representational and observational differences.
• Post-processing fairness calibration methods for regression are generally inapplicable in gaze estimation, as they require both sensitive attribute and image contexts and are not well-adapted to spatial prediction tasks.

The need for fundamentally new bias mitigation approaches—potentially exploiting representation disentanglement, causal learning, adversarial debiasing, or environment-invariant modeling—is highlighted.

Limitations and Open Directions

The paper sets a rigorous baseline but is limited to ethnicity and gender; age and other continuous sensitive variables remain unaddressed. Further, only datasets with adequate image quality and label certainty are considered, potentially underestimating real-world confounding. Post-processing fairness techniques for spatial regression tasks like gaze remain an open problem. The adoption of intersectional and contextual fairness metrics and joint demographic/environmental annotation is suggested as future research.

Conclusion

This work establishes the first fairness benchmark for appearance-based gaze estimation, providing demographic annotations, empirical evidence of severe bias in SOTA models, and a systematic evaluation of classical bias mitigation algorithms. The results highlight significant fairness deficits across ethnicity and gender, and the limited efficacy of standard rebalancing or reweighting strategies. Achieving fair and robust gaze estimation will require the design of architectures and mitigation algorithms specifically tailored to disentangle and correct for latent demographic- and environment-driven discrepancies. This paper provides essential infrastructure and exposes critical open challenges for the community.