FairLogue: A Toolkit for Intersectional Fairness Analysis in Clinical Machine Learning Models

Abstract: Objective: Algorithmic fairness is essential for equitable and trustworthy machine learning in healthcare. Most fairness tools emphasize single-axis demographic comparisons and may miss compounded disparities affecting intersectional populations. This study introduces Fairlogue, a toolkit designed to operationalize intersectional fairness assessment in observational and counterfactual contexts within clinical settings. Methods: Fairlogue is a Python-based toolkit composed of three components: 1) an observational framework extending demographic parity, equalized odds, and equal opportunity difference to intersectional populations; 2) a counterfactual framework evaluating fairness under treatment-based contexts; and 3) a generalized counterfactual framework assessing fairness under interventions on intersectional group membership. The toolkit was evaluated using electronic health record data from the All of Us Controlled Tier V8 dataset in a glaucoma surgery prediction task using logistic regression with race and gender as protected attributes. Results: Observational analysis identified substantial intersectional disparities despite moderate model performance (AUROC = 0.709; accuracy = 0.651). Intersectional evaluation revealed larger fairness gaps than single-axis analyses, including demographic parity differences of 0.20 and equalized odds true positive and false positive rate gaps of 0.33 and 0.15, respectively. Counterfactual analysis using permutation-based null distributions produced unfairness ("u-value") estimates near zero, suggesting observed disparities were consistent with chance after conditioning on covariates. Conclusion: Fairlogue provides a modular toolkit integrating observational and counterfactual methods for quantifying and evaluating intersectional bias in clinical machine learning workflows.

• The paper introduces a modular Python toolkit that integrates observational and counterfactual methods to assess intersectional fairness in clinical machine learning.
• The toolkit extends traditional fairness metrics like Demographic Parity and Equalized Odds to reveal compounded disparities, as shown in a glaucoma surgery prediction study.
• Empirical results indicate that while significant subgroup gaps exist, counterfactual analyses reveal these may stem from inherent data structure rather than explicit model bias.

FairLogue: A Toolkit for Intersectional Fairness Analysis in Clinical Machine Learning

Introduction

Fairness in algorithmic decision-making is essential for deploying ML models in clinical settings. Traditional tools for fairness analysis typically focus on single protected attributes and often fail to detect intersectional disparities—inequities uniquely affecting individuals grouped by the intersection of multiple demographic factors. "FairLogue: A Toolkit for Intersectional Fairness Analysis in Clinical Machine Learning Models" (2604.04858) introduces a modular, Python-based toolkit for comprehensive, intersectional fairness assessments integrating both observational and counterfactual analytic paradigms. This toolkit is specifically tailored for practical deployment in clinical ML workflows, as demonstrated in a real-world case study using the All of Us dataset for glaucoma surgery prediction.

Methodological Framework

FairLogue comprises three components, each corresponding to a distinct method for quantifying intersectional bias.

Component 1: Observational Intersectional Fairness Assessment

Component 1 generalizes established fairness metrics—Demographic Parity (DP), Equalized Odds (EO), and Equal Opportunity Difference (EOD)—to the intersectional case, enabling quantification of group-level disparities without imposing causal assumptions. The workflow requires input data with at least two protected attributes, a specified model (e.g., logistic regression, random forest), and the desired metrics. It outputs disaggregated group-wise DP, EO, and EOD, supporting both single-axis and intersectional analyses.

Figure 1: Component 1 workflow for observational intersectional fairness assessment, quantifying disparities for single and intersectional protected groups.

This design surfaces disparities that remain obscured in traditional, attribute-centric evaluations. Model hyperparameters and representation thresholds are tunable, accounting for small subgroup instability.

Component 2: Counterfactual Intersectional Fairness under Treatment

Component 2 adopts a counterfactual perspective, estimating fairness by evaluating group disparities under hypothetical reassignment of intersectional group membership while holding treatment and covariates fixed. The computational strategy, adapted from [wastvedt2023], leverages permutation-based null distributions and bootstrapping to quantify whether observed disparities in error rates (e.g., cFPR, cFNR) exceed those expected by chance.

Component 3: Generalized Counterfactual Intersectional Fairness

Component 3 extends counterfactual fairness analysis to arbitrary ML models and, crucially, to contexts without explicit treatment assignments. Identity group membership itself acts as the intervention, and the framework remains agnostic to treatment structure. The estimation strategy supports both single-robust and doubly-robust options (the latter via augmented inverse probability weighting) for enhanced robustness when the relationship between group membership and outcome varies. Observed group-level error rates are compared to permutation-based nulls, yielding $u$-values quantifying deviations from fairness under random assignment.

Figure 2: Workflow for counterfactual intersectional fairness analysis, generating permutation-based null distributions and calculating $u$-values for disparities persisting beyond chance.

Empirical Results: Glaucoma Surgery Prediction

The toolkit's utility is demonstrated on EHR data from the All of Us Controlled Tier V8 dataset, targeting the prediction of glaucoma surgery within six months post-diagnosis using a logistic regression model. Race and gender jointly define intersectional subgroups, with the main analysis cohort comprising 3,880 participants predominantly from four race-gender intersections. The cohort's outcome-positive rate (surgery within 6 months) was 18.7%.

Component 1 revealed significant intersectional fairness gaps despite moderate overall discriminatory performance (AUROC = 0.709, accuracy = 0.651). Specifically, intersectional evaluation identified a demographic parity gap of 0.20, EO true positive rate and false positive rate gaps of 0.33 and 0.15, respectively—magnitudes exceeding those found in analyses restricted to a single axis (DP gap for race: 0.10; for gender: 0.07; EO TPR gap for race: 0.08).

These disparities, shown by the difference in model sensitivity and selection rates across intersectional groups, highlight compounded unfairness that would not be detected in standard single-axis evaluations. However, observational analysis alone cannot attribute these disparities to model bias as opposed to data structure.

Component 3 was then applied, simulating counterfactual group assignments via permutation (SR estimation with bootstrapping, 200 null iterations). Resulting $u$-values across aggregate disparity metrics were near zero under an acceptability threshold of 0.1, indicating the observed disparities were statistically indistinguishable from those expected under chance. Thus, conditional on covariates, intersectional group identity did not systematically drive prediction errors.

Figure 3: Aggregate disparity metrics overlaid with $u$-values (red); a threshold of 0.1 demarcates acceptable unfairness.

Discussion

FairLogue's multipronged architecture exposes critical shortcomings of fairness auditing limited to single attributes. The case study demonstrates that intersectional analyses diagnose larger disparities than single-axis metrics, empirically confirming that traditional evaluations can underestimate inequities among intersecting marginalized groups. Such overlooked disparities have direct translational consequences, potentially exacerbating structural inequities in health care access and outcomes.

The integration of observational and counterfactual modules enables disambiguation between disparities reflectively arising from model architecture versus those due to underlying data associations. In the presented task, counterfactual analysis shows that observed disparities may not be attributable to model unfairness per se, but possibly to structural patterns in the data itself—a nuanced but essential distinction for intervention design.

Interpretation of counterfactual metrics requires attention: when correlations between protected attributes and outcome are weak, $u$-values will be indistinguishable from zero, reflecting either true fairness or a lack of statistical power. Thus, counterfactual analyses should be interpreted in conjunction with descriptive group-level disparities.

Practically, toolkit usability is enhanced via modular, well-documented code compatible with common ML frameworks. Subgroup sparsity and covariance mis-specification remain limiting factors, but mitigation options and diagnostic advice are provided. The methodology generalizes to other domains where intersectional fairness is a concern, supporting adoption beyond healthcare.

Conclusion

FairLogue operationalizes intersectional fairness auditing for clinical ML by synthesizing descriptive and counterfactual methodologies into an accessible, modular toolkit. Empirical application illustrates the capacity to uncover compounded disparities while contextualizing their origins, supporting robust, reproducible fairness evaluation. Available as open-source software, FairLogue lowers barriers for integration into clinical and other sensitive-domain ML pipelines. Future research may expand on the estimation properties for small samples or adapt the toolkit for dynamic, real-time auditing scenarios.