Mitigating class imbalance in the SAFARI benchmark

Determine how to mitigate the imbalance issue arising from SAFARI’s empirical distribution of real-world industrial HARA cases, including its uneven representation of risk categories.

Background

SAFARI preserves the empirical distribution of industrial HARA records rather than balancing the dataset artificially. Because real safety-engineering practice contains rare but safety-critical situations, the resulting benchmark has imbalanced risk-category distributions. The paper explicitly identifies mitigation of this imbalance as unresolved future development, while noting that the imbalance does not undermine the benchmark’s purpose of reflecting real-world industrial conditions.

References

How to mitigate the imbalance issue remains an open problem for future development, but it does not diminish our contribution for providing a benchmark that reflect real-world industrial distribution.

— SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment  (2609.20584 - Wu et al., 17 Sep 2026) in Section Limitations