Holistic Fairness Score Overview

• Holistic fairness score is a multidimensional metric that aggregates various fairness aspects using geometric means to reflect both intrinsic diversity and conditional robustness.
• It applies to areas like text-to-image evaluation, language model bias, and certified AI robustness by combining disparate fairness metrics into a single interpretable number.
• The design explicitly penalizes failure in any fairness dimension and provides supplementary diagnostics, enabling nuanced bias identification and mitigation.

Searching arXiv for the cited works to ground the article in current papers. arxiv_search: search_query="(Chen et al., 23 May 2026) HoloFair Unified T2I Fairness Evaluation and Fair-GRPO Debiasing", max_results=5 In recent machine-learning literature, a “holistic fairness score” denotes a single scalar that aggregates multiple fairness-relevant dimensions rather than reporting only a single disparity statistic or a single average utility. The expression has been used for text-to-image evaluation, certified robustness, LLM serving, AI-system certification, language-model bias benchmarking, cross-domain computer vision, and speech recognition, with each formulation combining several quantities into one score while retaining auxiliary diagnostics such as per-group disparities, worst-case behavior, or per-attribute bias indices (Chen et al., 23 May 2026).

1. Conceptual definition and recurrent mathematical forms

Holistic fairness scores are typically constructed to prevent one favorable dimension from obscuring another unfavorable one. In HoloFair, the Holistic Fairness Score is the Multi-attribute, Group-wise Bias Index (MGBI), defined as the geometric mean of an intrinsic diversity term and a context-robust conditional diversity term: $$\mathrm{MGBI}(G)=\sqrt{\max(\epsilon,R_{\mathrm{intrinsic}}(G))\cdot \max(\epsilon,R_{\mathrm{cond}}(G))}.$$ Both components lie in $[0,1]$, and the geometric mean ensures that a low score in one cannot be “made up” by a high score in the other; no additional weights are introduced, i.e. $1{:}1$ balance (Chen et al., 23 May 2026).

Other works instantiate the same aggregation idea with different operators. The Fairness Score for AI-system certification is

$$FS = 1-\sqrt{\frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}(M_{ij}-M'_j)^2},$$

compressing multiple fairness metrics across multiple protected attributes into one interpretable score in $[0,1]$ (Agarwal et al., 2022). GF-Score recommends a fairness-aware robustness summary through

$$\mathrm{FP\text{-}GREAT}(f;\lambda)=\bar{\Omega}(f)-\lambda\,\mathrm{RDI}(f),$$

and also gives a more general linear aggregate

$$\mathrm{HFS}(f)=w_1\bar{\Omega}(f)-w_2\mathrm{RDI}(f)+w_3\mathrm{WCR}(f)-w_4\mathrm{NRGC}(f),$$

with $\sum_i w_i=1$, $w_i\ge 0$ (Shah et al., 14 Apr 2026). In Equinox, the Holistic Fairness score for a user is

$$HF_f=\alpha\cdot UFC_f+\beta\cdot RFC_f,\qquad \alpha+\beta=1,$$

combining user-centric and system-centric counters for proactive fairness-aware scheduling (Wei et al., 19 Aug 2025). SAGED defines

$[0,1]$0

with default $[0,1]$1, so that higher means more fair after normalization and calibration (Guan et al., 2024). Harmonic Fairness in cross-domain computer vision instead uses the harmonic mean of shifted improvements in disparity and top-group accuracy: $[0,1]$2 which yields $[0,1]$3 at parity with the baseline and $[0,1]$4 under joint improvement (Iurada et al., 2023). ASR-FAIRBENCH merges fairness and accuracy through

$[0,1]$5

where $[0,1]$6 is an overall fairness score derived from a mixed-effects Poisson regression over WER (Rai et al., 16 May 2025).

This variety suggests that “holistic fairness score” is not a single canonical metric. It is a design family whose defining feature is multidimensional aggregation under an explicit trade-off rule.

2. HoloFair’s MGBI: formal construction for text-to-image fairness

HoloFair introduces a comprehensive benchmark framework for multidimensional demographic bias analysis in text-to-image models and defines its holistic fairness score as the Multi-attribute, Group-wise Bias Index. The attributes are $[0,1]$7, with demographic taxonomy $[0,1]$8, $[0,1]$9, and $1{:}1$0 (Chen et al., 23 May 2026).

For a neutral prompt $1{:}1$1 and a semantic trigger set $1{:}1$2 with $1{:}1$3 adjectives from the Stereotype Content Model, the framework generates $1{:}1$4 images per prompt and obtains empirical distributions

$1{:}1$5

With $1{:}1$6, the normalized entropy for attribute $1{:}1$7 under distribution $1{:}1$8 is

$1{:}1$9

The intrinsic diversity component is

$$FS = 1-\sqrt{\frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}(M_{ij}-M'_j)^2},$$0

This quantity measures neutral-prompt diversity across the demographic attributes.

The context-robust conditional diversity component is defined by first computing the per-trigger geometric mean

$$FS = 1-\sqrt{\frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}(M_{ij}-M'_j)^2},$$1

and then taking the lower $$FS = 1-\sqrt{\frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}(M_{ij}-M'_j)^2},$$2-quantile over $$FS = 1-\sqrt{\frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}(M_{ij}-M'_j)^2},$$3, with $$FS = 1-\sqrt{\frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}(M_{ij}-M'_j)^2},$$4 by default: $$FS = 1-\sqrt{\frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}(M_{ij}-M'_j)^2},$$5 This construction targets near-worst-case conditional behavior under semantic triggers.

The final MGBI is therefore

$$FS = 1-\sqrt{\frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}(M_{ij}-M'_j)^2},$$6

All entropies are normalized by $$FS = 1-\sqrt{\frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}(M_{ij}-M'_j)^2},$$7, and the entropy floor at $$FS = 1-\sqrt{\frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}(M_{ij}-M'_j)^2},$$8 avoids degeneracy. Compared to deviation-ratio metrics, MGBI’s entropy-based formulation severely penalizes mode collapse, described in the paper as minority erasure, and quantile aggregation highlights near-worst-case failure modes (Chen et al., 23 May 2026).

3. Dataset and SpaFreq classifier in the HoloFair pipeline

HoloFair’s holistic fairness score depends on a fairness-oriented dataset and the SpaFreq (Spatial-Frequency) attribute classifier. The prompt sets are all single-subject templates. The Gen set has approximately 300 prompts covering every $$FS = 1-\sqrt{\frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}(M_{ij}-M'_j)^2},$$9 combination and is used to generate images for classifier training. The Eval set has 750 prompts, consisting of 300 neutral prompts and 450 semantic triggers, and the Train set contains 10 K prompts for debiasing, strictly paraphrased to avoid overlap with Eval (Chen et al., 23 May 2026).

The Refactoring Bias Dataset for classifier training contains approximately 90 K real faces from FairFace+UTKFace balanced across $[0,1]$0–$[0,1]$1, approximately 2 K in-the-wild portraits to improve robustness, and approximately 20 K synthetic portraits from eight T2I models to close domain gap. Annotation and quality control follow a three-stage pipeline—automatic CLIPScore plus YOLOv8 face-filtering, then LVLM majority voting, then human adjudication—yielding high-precision labels for gender, age, and race (Chen et al., 23 May 2026).

SpaFreq uses DINOv2-Base as a backbone encoding CLS tokens. For each image $[0,1]$2, it constructs a dual-stream input: a spatial view $[0,1]$3 given by the original RGB image, and a frequency view $[0,1]$4 given by a 3-channel wavelet transform using $[0,1]$5, $[0,1]$6, and $[0,1]$7, with $[0,1]$8 discarded and per-band min-max normalization followed by channel concatenation. The two views are stacked along the batch dimension,

$[0,1]$9

then split after the backbone into $$\mathrm{FP\text{-}GREAT}(f;\lambda)=\bar{\Omega}(f)-\lambda\,\mathrm{RDI}(f),$$0. A learnable fusion weight $$\mathrm{FP\text{-}GREAT}(f;\lambda)=\bar{\Omega}(f)-\lambda\,\mathrm{RDI}(f),$$1, initialized at $$\mathrm{FP\text{-}GREAT}(f;\lambda)=\bar{\Omega}(f)-\lambda\,\mathrm{RDI}(f),$$2, defines

$$\mathrm{FP\text{-}GREAT}(f;\lambda)=\bar{\Omega}(f)-\lambda\,\mathrm{RDI}(f),$$3

and the final feature is

$$\mathrm{FP\text{-}GREAT}(f;\lambda)=\bar{\Omega}(f)-\lambda\,\mathrm{RDI}(f),$$4

which is passed to an MLP head for attribute logits (Chen et al., 23 May 2026).

The classifier is trained end-to-end on RBD with AdamW, learning rate $$\mathrm{FP\text{-}GREAT}(f;\lambda)=\bar{\Omega}(f)-\lambda\,\mathrm{RDI}(f),$$5, weight decay $$\mathrm{FP\text{-}GREAT}(f;\lambda)=\bar{\Omega}(f)-\lambda\,\mathrm{RDI}(f),$$6, batch size $$\mathrm{FP\text{-}GREAT}(f;\lambda)=\bar{\Omega}(f)-\lambda\,\mathrm{RDI}(f),$$7, for $$\mathrm{FP\text{-}GREAT}(f;\lambda)=\bar{\Omega}(f)-\lambda\,\mathrm{RDI}(f),$$8 epochs. It achieves approximately $$\mathrm{FP\text{-}GREAT}(f;\lambda)=\bar{\Omega}(f)-\lambda\,\mathrm{RDI}(f),$$9 gender, $$\mathrm{HFS}(f)=w_1\bar{\Omega}(f)-w_2\mathrm{RDI}(f)+w_3\mathrm{WCR}(f)-w_4\mathrm{NRGC}(f),$$0 age, and $$\mathrm{HFS}(f)=w_1\bar{\Omega}(f)-w_2\mathrm{RDI}(f)+w_3\mathrm{WCR}(f)-w_4\mathrm{NRGC}(f),$$1 race accuracy on a held-out synthetic face test. These details matter because MGBI is only as reliable as the demographic distributions estimated by this classifier.

4. Empirical behavior of MGBI and the relation to Fair-GRPO debiasing

HoloFair evaluates eight popular T2I models: SDXL, SD3.5-L, Flux1-dev, SANA-1.5, Show-o, Harmon, Bagel, and Blip3-o. The reported results illustrate the intended behavior of MGBI under both neutral and bias-triggering prompts (Chen et al., 23 May 2026).

Model	Reported pattern	MGBI
SDXL	highest $$\mathrm{HFS}(f)=w_1\bar{\Omega}(f)-w_2\mathrm{RDI}(f)+w_3\mathrm{WCR}(f)-w_4\mathrm{NRGC}(f),$$2, but $$\mathrm{HFS}(f)=w_1\bar{\Omega}(f)-w_2\mathrm{RDI}(f)+w_3\mathrm{WCR}(f)-w_4\mathrm{NRGC}(f),$$3	$$\mathrm{HFS}(f)=w_1\bar{\Omega}(f)-w_2\mathrm{RDI}(f)+w_3\mathrm{WCR}(f)-w_4\mathrm{NRGC}(f),$$4
Flux1-dev	more balanced tail performance, $$\mathrm{HFS}(f)=w_1\bar{\Omega}(f)-w_2\mathrm{RDI}(f)+w_3\mathrm{WCR}(f)-w_4\mathrm{NRGC}(f),$$5	$$\mathrm{HFS}(f)=w_1\bar{\Omega}(f)-w_2\mathrm{RDI}(f)+w_3\mathrm{WCR}(f)-w_4\mathrm{NRGC}(f),$$6
Show-o	more balanced tail performance, $$\mathrm{HFS}(f)=w_1\bar{\Omega}(f)-w_2\mathrm{RDI}(f)+w_3\mathrm{WCR}(f)-w_4\mathrm{NRGC}(f),$$7	$$\mathrm{HFS}(f)=w_1\bar{\Omega}(f)-w_2\mathrm{RDI}(f)+w_3\mathrm{WCR}(f)-w_4\mathrm{NRGC}(f),$$8
Blip3-o	worst fairness	$$\mathrm{HFS}(f)=w_1\bar{\Omega}(f)-w_2\mathrm{RDI}(f)+w_3\mathrm{WCR}(f)-w_4\mathrm{NRGC}(f),$$9

These numbers show why the metric combines intrinsic and conditional components multiplicatively rather than additively. SDXL had the highest intrinsic diversity but suffered a collapse under biased prompts, so its final score was lower than systems with weaker neutral diversity but stronger tail robustness. Bootstrap $\sum_i w_i=1$0 confidence intervals on $\sum_i w_i=1$1, $\sum_i w_i=1$2, and $\sum_i w_i=1$3 show that the ranking is stable across $\sum_i w_i=1$4 (Chen et al., 23 May 2026).

Beyond evaluation, HoloFair introduces Fair-GRPO, a reinforcement-learning-based debiasing method that alters the distribution of generative models through a designed multi-objective reward function. On SD3.5-Medium, Fair-GRPO significantly improves multidimensional fairness while maintaining high image quality. The work also analyzes potential reward hacking phenomena and provides corresponding mitigation strategies (Chen et al., 23 May 2026).

A recurring misconception in fairness evaluation is that a model appearing diverse under neutral prompts is thereby fair. HoloFair directly rejects that implication: intrinsic diversity and robustness to semantic triggers are separated, and the final score penalizes failure in either dimension. Another misconception is that a single average fairness value suffices; the HoloFair design keeps $\sum_i w_i=1$5 and $\sum_i w_i=1$6 explicit so that “default” unfairness and “contextual” unfairness can diverge during development (Chen et al., 23 May 2026).

5. Related holistic fairness scores across domains

Outside text-to-image modeling, the same single-number aggregation principle has been specialized to several domains.

In certified robustness, GF-Score decomposes the GREAT Score into per-class guarantees

$\sum_i w_i=1$7

and quantifies inequality with four welfare-theoretic metrics: Robustness Disparity Index,

$\sum_i w_i=1$8

Normalized Robustness Gini Coefficient,

$\sum_i w_i=1$9

Worst-Case Class Robustness,

$w_i\ge 0$0

and FP-GREAT. The framework is intended to reward high overall robustness while penalizing large gaps between classes, ensuring that no class is left unprotected (Shah et al., 14 Apr 2026).

In LLM serving, Equinox separates user and operator perspectives. The User Fairness Counter updates as

$w_i\ge 0$1

with $w_i\ge 0$2 in Equinox, while the Resource Fairness Counter updates as

$w_i\ge 0$3

The scheduling policy always admits the next request from the user whose current $w_i\ge 0$4 is minimal, implementing max-min fairness (Wei et al., 19 Aug 2025).

In standardized auditing and certification, the Fairness Score aggregates deviations from ideal values across multiple metrics and protected attributes, while the Bias Index

$w_i\ge 0$5

reveals comparative bias amongst the various protected attributes within the dataset. The standard operating procedure culminates in computing $w_i\ge 0$6 and $w_i\ge 0$7, then issuing a Fairness Certificate if all individual metrics stayed within band and $w_i\ge 0$8 (Agarwal et al., 2022).

In language-model bias benchmarking, SAGED combines impact ratio, max disparity, and max Z-score after baseline calibration

$w_i\ge 0$9

and counterfactual branching. This explicitly treats metric tool bias and contextual prompt bias as first-order concerns in score construction (Guan et al., 2024).

In cross-domain computer vision, Harmonic Fairness measures improvement relative to a chosen baseline through disparity reduction and top-group accuracy improvement. Because it uses a harmonic mean, a model cannot “cheat” by boosting only accuracy or only fairness; $$HF_f=\alpha\cdot UFC_f+\beta\cdot RFC_f,\qquad \alpha+\beta=1,$$0 corresponds to no change from the baseline, and $$HF_f=\alpha\cdot UFC_f+\beta\cdot RFC_f,\qquad \alpha+\beta=1,$$1 indicates joint improvement (Iurada et al., 2023).

In speech recognition, ASR-FAIRBENCH fits a mixed-effects Poisson regression to utterance-level WER, converts subgroup coefficients into a 0–100 fairness scale, discounts category scores by a likelihood-ratio-test factor when $$HF_f=\alpha\cdot UFC_f+\beta\cdot RFC_f,\qquad \alpha+\beta=1,$$2, pools them into an overall fairness score $$HF_f=\alpha\cdot UFC_f+\beta\cdot RFC_f,\qquad \alpha+\beta=1,$$3, and combines fairness with accuracy through FAAS. The framework reports that Whisper-medium achieves the highest FAAS, approximately $$HF_f=\alpha\cdot UFC_f+\beta\cdot RFC_f,\qquad \alpha+\beta=1,$$4, while HuBERT-large and Wav2Vec-large rank lower due to fairness penalties despite comparable average WER (Rai et al., 16 May 2025).

6. Interpretation, limitations, and recurrent controversies

A central methodological issue is what exactly a holistic fairness score should aggregate. HoloFair’s MGBI targets marginal balance over gender, age, and race, but joint intersectional fairness $$HF_f=\alpha\cdot UFC_f+\beta\cdot RFC_f,\qquad \alpha+\beta=1,$$5 is not directly enforced; discrete categories and the single-subject restriction omit intersectional and multi-person scenarios, and the five-way race grouping merges Middle Eastern and Latino/Hispanic for classifier reliability (Chen et al., 23 May 2026). This makes clear that a one-number score can be principled without being exhaustive.

The certification-oriented Fairness Score has a different limitation profile. It is useful for benchmarking, comparison, and a clear pass/fail threshold, but its meaning depends on the chosen metrics, their ideal targets, and the tolerance band. The framework itself notes that if $$HF_f=\alpha\cdot UFC_f+\beta\cdot RFC_f,\qquad \alpha+\beta=1,$$6 is unavailable to the auditor, pre-training checks are skipped and the limitation should be noted in the certificate (Agarwal et al., 2022).

GF-Score highlights a different controversy: a model with higher average certified robustness may exhibit greater class-level disparity. The framework reports that more robust models tend to exhibit greater class-level disparity, which means a robustness leaderboard can conflict with a fairness-aware ranking if only mean guarantees are used (Shah et al., 14 Apr 2026).

Cross-domain Harmonic Fairness addresses yet another failure mode of standard metrics. Difference in Equal Opportunity, Difference in Equalized Odds, and Difference in Accuracy only capture subgroup discrepancies and can be driven to zero by a trivial classifier, while minimum-group accuracy alone does not reveal whether improvement came from leveling down the best-performing group. The baseline-relative harmonic form was proposed precisely to penalize such lopsided trade-offs (Iurada et al., 2023).

Causality-based fairness brings identifiability to the forefront. PC-Fairness formulates path-specific counterfactual fairness as a unified definition and shows that, under unidentifiable settings, one can still compute tight lower and upper bounds on the path-specific counterfactual effect through a linear program over response-function variables. A plausible implication is that any holistic fairness score built from counterfactual notions inherits the identifiability constraints of the underlying causal estimand rather than escaping them by aggregation (Wu et al., 2019).

Across these formulations, the principal methodological lesson is consistent: holistic fairness scores are most informative when reported together with their components, calibration assumptions, and failure modes. The single number is a summary, not a substitute for the underlying disparity structure.