Post-correction evaluation criteria for PRA models reviewed by AI

Establish criteria to judge the quality and acceptability of a Probabilistic Risk Assessment (PRA) model after all errors detected by a generative AI reviewer have been corrected, including how such corrections should influence model validation, acceptance, and potential regulatory review.

Background

The authors note that AI can flag errors and inconsistencies in PRA models, but there is no established methodology for evaluating the model once all AI-identified issues have been addressed.

Clarifying post-correction evaluation criteria is necessary to determine whether AI-assisted revisions produce models that meet quality and regulatory standards, and to define the role of AI outputs in formal review processes.

References

The following questions remains to be answered: what is the minimum error detection rate to leverage the advantages of the technology? What is the admissible false positive error detection rate? How do we judge a model where all AI detected errors were fixed?

— Impact of Generative AI (Large Language Models) on the PRA model construction and maintenance, observations (2406.01133 - Rychkov et al., 3 Jun 2024) in Observation 3, Section 3 (A generative AI use case for a fault tree review)