Dice Question Streamline Icon: https://streamlinehq.com

Post-correction evaluation criteria for PRA models reviewed by AI

Establish criteria to judge the quality and acceptability of a Probabilistic Risk Assessment (PRA) model after all errors detected by a generative AI reviewer have been corrected, including how such corrections should influence model validation, acceptance, and potential regulatory review.

Information Square Streamline Icon: https://streamlinehq.com

Background

The authors note that AI can flag errors and inconsistencies in PRA models, but there is no established methodology for evaluating the model once all AI-identified issues have been addressed.

Clarifying post-correction evaluation criteria is necessary to determine whether AI-assisted revisions produce models that meet quality and regulatory standards, and to define the role of AI outputs in formal review processes.

References

The following questions remains to be answered: what is the minimum error detection rate to leverage the advantages of the technology? What is the admissible false positive error detection rate? How do we judge a model where all AI detected errors were fixed?

— Impact of Generative AI (Large Language Models) on the PRA model construction and maintenance, observations (2406.01133 - Rychkov et al., 3 Jun 2024) in Observation 3, Section 3 (A generative AI use case for a fault tree review)