Evaluations that sufficiently cover model vulnerabilities

Determine methods and criteria to assess whether an evaluation procedure has identified most vulnerabilities of a given AI system, especially for capabilities that could enable harmful misuse or make systems difficult to oversee or control.

Background

The paper highlights that behavioral evaluations can miss important failure modes, providing at best lower bounds on harmful capabilities. This undermines confidence in safety assessments, particularly for high-risk behaviors such as deception or long-horizon planning.

A clear methodology to judge the thoroughness and sufficiency of evaluations would enable stronger assurance and better governance decisions in settings where failure costs are high.

References

Determining whether an evaluation procedure has identified all, if not most, of the vulnerabilities of a system is an open problem.

— Open Problems in Technical AI Governance  (2407.14981 - Reuel et al., 2024) in Section 3.3.1 Reliable Evaluations

With the disconfirming review always visible, the hit rate sits at ceiling, so the no-gullibility result leaves open whether a trace can mask a genuinely concealed error; a condition with buried errors is the natural test.

— Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers  (2609.18204 - Chen et al., 16 Sep 2026) in Section 6.3, Limitations and future work

What remains open is how to explore this space under a limited evaluation budget while prioritising failures that are both severe and high-fidelity.

— TESTNAV: Pareto-Guided Search for Compositional Robustness Testing  (2608.19882 - Arif et al., 20 Aug 2026) in Section 2.3, “Configuration-level compositional testing”

The repair does not restore adequacy, and the diagnosis is not exhaustive. After repair the observed network still falls outside its reachability reference for the majority of sub-samples in all four cells and the head-concentration and community blocks continue to fail, the median real distance moving from 2.1 to 2.9 times the threshold at baseline to 1.2 to 1.7 times after repair; and the statistic-held-out audit found a buyer-breadth-dispersion failure the diagnose-and-localize step had not flagged, so we do not claim the procedure identifies every misspecification.

— Testing, not presuming, adequacy: calibrating generative social simulators against emergent network structure  (2609.24012 - Shao et al., 21 Sep 2026) in Limitations, “The repair does not restore adequacy, and the diagnosis is not exhaustive”

Evaluation should compare distinct reproducible discoveries at matched search budgets, while testing saved states under overlapping evaluator versions. This would help distinguish wider coverage from a stricter judge, and system improvement from adaptation to the test. The degree of independence needed between generator, actor, and judge remains unresolved.

— Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation  (2609.31186 - Gong et al., 25 Sep 2026) in Section 6, subsection “Safety Evaluation under a Changing Judge”

Adaptive evaluations can find failures that fixed tests miss. The space of unknown failures has no observable denominator. The problem becomes even harder under partial observability, where evaluators may see only fragments of the system state, update history, or downstream consequences.

— Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation  (2609.31186 - Gong et al., 25 Sep 2026) in Section 6, subsection “Safety Evaluation under a Changing Judge”

Furthermore, it is unclear how well honeypots can distinguish between malicious and benign misbehaviors.

— Honeypots for AI Agent Insider Threats  (2610.26664 - Wehner et al., 19 Oct 2026) in Section 3, “Empirical Evidence for Incrimination”; Section 8, “Technical Research Agenda”