Auditing Demographic Bias in Facial Landmark Detection for Fair Human-Robot Interaction

Abstract: Fairness in human-robot interaction critically depends on the reliability of the perceptual models that enable robots to interpret human behavior. While demographic biases have been widely studied in high-level facial analysis tasks, their presence in facial landmark detection remains unexplored. In this paper, we conduct a systematic audit of demographic bias in this task, analyzing the age, gender and race biases. To this end we introduce a controlled statistical methodology to disentangle demographic effects from confounding visual factors. Evaluations of a standard representative model demonstrate that confounding visual factors, particularly head pose and image resolution, heavily outweigh the impact of demographic attributes. Notably, after accounting for these confounders, we show that performance disparities across gender and race vanish. However, we identify a statistically significant age-related effect, with higher biases observed for older individuals. This shows that fairness issues can emerge even in low-level vision components and can propagate through the HRI pipeline, disproportionately affecting vulnerable populations. We argue that auditing and correcting such biases is a necessary step toward trustworthy and equitable robot perception systems.

• The paper demonstrates that confounding visual factors like head pose and face resolution explain up to 51.5% of the normalized mean error in landmark detection.
• The paper shows that observed biases in gender and race vanish when controlling for confounders, emphasizing the need for robust auditing strategies.
• The paper identifies a persistent age bias driven by severe sample imbalance, underlining the urgency for more demographically diverse training datasets.

Auditing Demographic Bias in Facial Landmark Detection in Human-Robot Interaction

Motivation and Context

This work systematically investigates demographic bias in facial landmark detection, a core low-level component that supports perception pipelines in human-robot interaction (HRI). Historically, fairness research in face analysis is concentrated on downstream tasks such as face recognition, gender classification, and affective analysis, with limited attention to the foundational step of landmark localization. Given that landmark estimation accuracy directly impacts high-level modules—expression recognition, gaze tracking, attention estimation—identifying the presence and magnitude of demographic biases in this stage is critical for equitable and trustworthy robot perception systems.

The principal research question is to what extent variations in facial landmark detector performance can be genuinely ascribed to user demographic attributes (gender, age, race), considering the substantial confounding effects of physical and visual factors (head pose, image resolution, occlusion, expression). The analysis is motivated by the unique risks in HRI, where algorithmic failures can be misattributed to the human user due to the "veil of objectivity" that surrounds embodied agents.

Methodological Framework

A representative U-Net-based facial landmark detection model with a ResNet-34 encoder was selected as the backbone. Training employed the WFLW dataset (98-point annotation), while evaluation occurred on both WFLW and RAF-DB, the latter of which provides well-structured demographic metadata (age, gender, race) and five-point landmark annotations. For WFLW, demographic annotation was retrofitted using an ensemble approach leveraging publicly available classifiers to maximize consistency with RAF-DB's taxonomy.

To rigorously disentangle demographic effects from visual confounders, the study utilized linear regression models predicting normalized mean error (NME), the standard metric in facial landmarking, on a per-image basis. The explanatory variables included demographic attributes and multiple confounding factors: head pose (quantified by geodesic distance from frontal orientation via continuous Euler angles), face resolution (inverse of bounding box height), and, where available, annotations for expression, occlusion, blur, illumination, and makeup.

Regression residuals were transformed with Box–Cox for homoscedasticity. Statistical significance assessments leveraged Type III ANOVA, and category-level marginal means were computed with adjusted confidence intervals.

Empirical Findings

The primary empirical observations are as follows:

• Confounding Visual Factors Dominate: Head pose and face resolution explain a large fraction of the variance in NME (bounding box height alone explains over 29.3% in RAF-DB), with much higher effect sizes than any demographic variable. The regression models with all confounders accounted for up to 51.5% of the NME variance on RAF-DB.
• Gender and Race Bias Vanish When Controlling for Confounders: Apparent performance differentials by gender and race are explained by correlations with visual confounders (notably, head pose and image resolution, and to a lesser degree expression distributions). When these confounders are included in the regression, the statistical significance of gender and race effects is eliminated for both test sets.
• Pronounced Age Bias Remains: In RAF-DB, a substantial and statistically significant age effect persists, even when all confounders are controlled. Marginal NME for individuals over 69 is consistently and significantly higher. This pattern is not observed in WFLW, likely due to severe under-sampling (only 7 test images in "70+" in WFLW). The findings strongly implicate sample imbalance in training data as the key driver of age-related disparity.
• Expression and Other Attributes: Emotional expression impacts localization error, with greatest errors in "fear" and lowest in "happiness." Other visual attributes (e.g., occlusion, blur, makeup) also retain significant effects, albeit smaller than head pose and resolution.

  Figure 1: Distribution of images by expression category and gender in RAF-DB reveals interaction between demographic and behavioral attributes, implicated in apparent demographic disparity.

Implications and Discussion

The results deliver several critical implications for AI and robotics:

• Necessity of Controlled Audits: Traditional fairness assessments based on raw demographic-wise performance comparisons are liable to confounding and often lead to misinterpretation. Robust auditing must strictly control for visual factors that independently affect model error.
• Fairness Propagation in HRI Pipelines: Given the hierarchical utilization of landmark detection in HRI perception stacks, low-level age bias can propagate and be amplified by downstream modules, contributing to systematic underserving of elderly populations. Bias in the perceptual base layers cannot be compensated by high-level algorithmic corrections alone.
• Data Imbalance and Vulnerable Populations: The age bias observed tracks with severe data underrepresentation of older adults in common datasets (highlighted by only 43 "70+" samples in WFLW train). This finding underscores the importance of balanced dataset curation and motivates collection of new demographic-diverse training corpora to mitigate performance disparities for vulnerable users.
• Technological Recommendations: Improving landmark detector robustness under challenging head poses and low-resolution settings is likely to yield greater overall and equitable gains than optimizing for demographic-specific errors in isolation.

Theoretical Considerations and Future Directions

These findings reinforce the notion that observed "algorithmic biases" in perception modules are largely reflective of data sampling and environmental factors, rather than intrinsic model discrimination based on demographic identity. However, failure to acknowledge or audit these upstream biases can result in significant social harms when leveraged in interactive, high-stakes domains such as assistive robotics.

Future research pathways include:

• Extending the bias analysis to additional demographic axes (e.g., intersectional identities, disability status).
• Leveraging causal inference techniques to explicitly quantify mediation by confounding factors.
• Developing new augmentation and synthetic data generation techniques to address sample imbalance, and reweight training to target equitable performance.
• Exploring bias mitigation methods tailored for low-level vision tasks—given differing error structure compared to classification.

Conclusion

This study provides the first systematic disentanglement of demographic and visual confounding effects on facial landmark detector fairness in HRI settings. The results establish that the dominant sources of landmark error are physical and imaging variables—not demographic identity—except for a persistent age bias due to extreme data underrepresentation. The findings urge the field to go beyond black-box reliance on benchmarks, instituting rigorous auditing and dataset redesign to ensure perception equity, particularly for vulnerable user groups. Ensuring trustworthy human-robot interaction mandates both technical and epistemic vigilance throughout the entire perception stack.