Calibrate and empirically validate FHA impact and likelihood assessments

Develop calibration guidance and empirically evaluate the reliability and soundness of impact and likelihood assessments in Step C of Fairness Hazard Analysis for socio-technical processes.

Background

The multiple-case study used qualitative impact and likelihood ratings to assess fairness hazards, but did not provide detailed calibration guidelines or evaluate whether the resulting ratings were reliable or sound. The paper indicates that addressing this limitation would require sustained participation from experts in fairness, risk assessment, the application domain, and the organisational process, potentially using structured expert elicitation, Delphi methods, or nominal-group procedures. Such systematic calibration and empirical evaluation are deferred to later validation stages at TRL 6 and TRL 7.

References

In the current study, we do not provide detailed calibration guidelines for impact and likelihood assessment, nor do we evaluate the reliability or soundness of the resulting ratings. Establishing such guidance requires a dedicated study involving sustained participation from experts in fairness, risk assessment, the application domain, and the organisational process. Techniques such as structured expert elicitation, the Delphi method, or nominal-group procedures may be used to iteratively compare judgements, clarify disagreements, and develop shared assessment criteria under uncertainty. The limited availability of the required experts made such an evaluation infeasible at the present stage. We therefore defer the systematic calibration and empirical evaluation of Step~C to subsequent validation at TRL~6 and TRL~7.

— Fairness Hazard Analysis for Socio-Technical Processes: A Multiple-Case Study in Bias-sensitive Organisational Settings  (2608.22978 - Broccia et al., 24 Aug 2026) in Footnote in Section FHA Operationalisation in the Multiple-Case Study, Phase 4: Application of FHA